Harley-ml/lesswrong huggingface dataset:
# Multiple Deploy-Key Repos

Let's say you want to give a machine access to one of your repos but not all of them. This is a reasonable thing to want, and so GitHub offers [deploy keys](https://docs.github.com/en/developers/overview/managing-deploy-keys). But what if you want to give it access to two repos?

The natural thing to try would be to add the same public key to your second repo, but GitHub rejects this:

Error: Key is already in use

I think this error is because they don't want you to get into a situation where multiple machines are using the same key. If that happened and you needed to revoke one machine's access, you'd be stuck. In this case, however, we only have one machine and we're trying to use the same key for two repos. I don't see any issues with that setup, and while maybe I'm not being imaginative enough I think GitHub should probably be checking for duplicate deploy keys on a per-repo basis instead of globally?

Still, what can we do with GitHub as it is? Generate more keys and use aliases!

I'm going to walk through this assuming you're starting from scratch trying to check out both `github.com/you/repo1` and `github.com/you/repo2`. If you already have `repo1` working and don't mind having the two repos configured differently, just follow the `repo2` steps.

First, generate a ssh key for each repo:

$ ssh-keygen -t ed25519 -C "machineName-repo1"
Enter file in which to save the key: /home/you/.ssh/id_ed25519.repo1
...

$ ssh-keygen -t ed25519 -C "machineName-repo2"
Enter file in which to save the key: /home/you/.ssh/id_ed25519.repo2
...

Then visit `github.com/you/repo1/settings/keys` and `github.com/you/repo1/settings/keys` and paste the contents of `~/.ssh/id_ed25519.repo1` and `~/.ssh/id_ed25519.repo2` respectively.

At this point you've created the keys and told GitHub to respect them, but you haven't told `ssh` on your machine when to use which key. You do that in `~/.ssh/config`:

Host github-repo1
     HostName github.com
     User git
     IdentityFile ~/.ssh/id_ed25519.repo1
     IdentitiesOnly yes

Host github-repo2
     HostName github.com
     User git
     IdentityFile ~/.ssh/id_ed25519.repo1
     IdentitiesOnly yes

Anyway, now you can check out your repos:

$ git clone github-repo1:you/repo1.git
$ git clone github-repo2:you/repo2.git

The reason this works is that `git`, like [anything else](https://www.jefftk.com/p/mosh) that uses `ssh`, doesn't actually interpret the host name or set up the connection. It just asks `ssh` "please connect me to `github-repo1`" and `ssh` will use aliases as part of figuring out how to do that. This also means that almost any time you might have used `GIT_SSH_COMMAND`, or otherwise passed arguments to `ssh`, an alias is a better choice.

If you'd already checked out your repo, however, instead of checking it out again you just change where `origin` points:

$ cd repo1
$ git remote set-url origin github-repo1:you/repo1.git

And the same for `repo2`.
# [Video] How having Fast Fourier Transforms sooner could have helped with Nuclear Disarmament - Veritasium

Veritasium presents an argument that if we'd had Fast Fourier Transforms sooner, this would have made it possible to detect underground nuclear tests without on-site audits, somewhat better nuclear non-proliferation treaties could have been signed, and nuclear proliferation would have been slowed.

This was really interesting, I hadn't heard the story of these very early, hopeful, tragically failed attempts at nuclear disarmament. I'm troubled to learn that the Soviets rejected the Baruch Plan, measures analogous to [my geopolitical AGI non-proliferation story](https://forum.effectivealtruism.org/posts/kB2DK6fpmbGCxe4ED/appendix-to-a-safe-path). They saw on-site audits as "espionage". They saw this proposal to form an international nuclear authority as a threat to their sovereignty. To some extent, it could have been. There's a lot to chew on there.

A particularly interesting detail is that Fast Fourier Transforms actually *were* discovered earlier, but then forgotten, so no one can argue for fatalism here, it was not inevitable that we'd end up with this progress differential. Other histories were possible.
# Could a Supreme Court suit work to solve NEPA problems?

According to [Eli Dourado](https://www.thecgo.org/benchmark/much-more-than-you-ever-wanted-to-know-about-nepa/), the original interpretation of the NEPA law was reasonable. The earliest Environmental Impact Statements (EIS) were often less than ten pages. Afterward, through a combination of district court decisions and the decisions of the Council on Environmental Quality the work required to create an EIS ballooned to make it unreasonably expensive to comply with the EIS requirements.

It's noteworthy that neither laws passed by congress nor court decisions by the Supreme court are responsible in the growth of the work.

Whenever cases reached the Supreme Court, the court usually ruled against expansive interpretations of NEPA. Zachary Green describes the court decisions in his master thesis "[NEPA in the Supreme Court:  A history of defeat](https://repository.lib.ncsu.edu/bitstream/handle/1840.4/8624/Green%2C%20Zachary%20final.pdf?sequence=1&isAllowed=y)" by saying:

> Since its enactment in 1970, the National Environmental Policy Act (NEPA) has been the focus of seventeen Supreme Court cases. Government and industry have defeated environmental organizations in each of those seventeen cases.

In [United States v. Students Challenging Regulatory Agency Procedures](https://scholar.google.com/scholar_case?case=17168419660253988889&hl=en&as_sdt=2006) the Supreme Court argued:

> But [694*694](https://scholar.google.com/scholar_case?case=13398073460615865106#p694) nowhere, either in the legislative history or the statutory language, is there any indication that Congress intended to restore to the federal courts the power temporarily to suspend railroad rates, a power that had been clearly taken away by § 15 (7) of the Interstate Commerce Act.
> 
> The statutory language, in fact, indicates that NEPA was not intended to repeal by implication any other statute.

This is a clear statement against an overreaching reading of NEPA. Given this history of Supreme Court cases, it seems to me that there's a lot of room for the Supreme Court to rule against the current interpretation of NEPA without having to rule against any precedent. Given the current trend of ruling very pro-business, the current court would likely be even more willing than past courts to rule against expansive interpretations NEPA.

This year the Supreme Court decided that the [EPA](https://edition.cnn.com/2022/06/30/politics/supreme-court-climate-change-epa-regulations/index.html) overreached when they decided to regulate CO2. In the process they created precedent that reduces the amount of freedom agencies have to just do what they want.

One argument might be to say that the Council on Environmental Quality overreaches the authority that congress has given it?
# Mechanistic Interpretability as Reverse Engineering (follow-up to "cars and elephants")

I think (perhaps) the distinction that I was trying to make in my previous post ["Cars and Elephants": a handwavy argument/analogy against mechanistic interpretability](https://www.lesswrong.com/posts/YEkzeJTrp69DTn8KD/cars-and-elephants-a-handwavy-argument-analogy-against) is basically the distinction between **engineering** and **reverse engineering.  **

**Reverse engineering** is analogous to mechanistic interpretability; **engineering** is analogous to "well-founded AI" (to borrow [Stuart Russell's term](https://www.youtube.com/watch?v=mYOg8_iPpFg&feature=emb_logo&ab_channel=StanfordExistentialRisksInitiative)).  
  
So it seems worth exploring the pros and cons of these two approaches to understanding x-safety-relevant properties of advanced AI systems.    
  
As a gross simplification,[^zipb8g2097a] we could view the situation this way:

*   Using deep learning approaches, we can build advanced AI systems that are not well understood.  Better **reverse engineering** would make them better understood.
*   Using "well-founded AI" approaches, we can build AI systems that are well understood, but not as advanced.  Better **engineering** would make them more advanced. 

Under this view, these two approaches are working towards the same end from different starting points.    
  
A few more thoughts:

*   Competitiveness arguments favor **reverse engineering**.  Safety arguments favor **engineering**.
*   We don't have to choose one.  We can work from both ends, and look for ways to combine approaches.
*   I'm not sure which end is easier to start from.  My intuition says that there is the same underlying difficulty that needs to be addressed regardless of where you start from,[^yhmwbj2opql] but the perspective I'm presenting seems to suggest otherwise.
*   There may be some sort of P vs. NP kind of argument in favor of reverse engineering, but it seems likely to rely on some unverifiable assumptions (e.g. that we will in fact reliably recognize good mechanistic interpretations).

[^zipb8g2097a]:  I know people will say that we don't actually understand how "Well founded AI" approaches work any better.  I don't feel equipped to evaluate that claim beyond extremely simple cases, and don't expect most readers are either. 

[^yhmwbj2opql]: At least if your goal is to get something like an AGI system, the safety of which we have justified confidence in.  This is perhaps too ambitious of a goal.
# Metaculus is seeking Software Engineers

(See Metaculus's previous career-tagged posts [here](https://www.lesswrong.com/posts/bGueGuSsDpRPJF8aD/metaculus-is-seeking-analytical-storytellers-to-write-essays), [here,](https://www.lesswrong.com/posts/WaEd8oHHKD5AcYzBu/metaculus-is-seeking-experienced-leaders-researchers-and) and [here](https://www.lesswrong.com/posts/JqtPkFqmJtoHTahis/metaculus-is-building-a-team-dedicated-to-ai-forecasting).) tl;dr Metaculus is now hiring [Frontend Engineers](https://apply.workable.com/metaculus/j/B36C3B4BF1/) and [Full Stack Engineers.](https://apply.workable.com/metaculus/j/409AECAA94/)

Since 2015, [Metaculus](https://www.metaculus.com/questions/) has been a core part of the [epistemic infrastructure](https://metaculus.medium.com/forecasting-science-and-epistemology-d962de541a1f) at the heart of the LW/OB/EA world. I'm posting here because I myself was just hired at Metaculus in part due to one of the above linked posts advertising to LW.

I expect many of you have been following the forecasting space for a long time. We're at an inflection point now, especially at Metaculus as evident from our recent funding and partnerships, but also at a variety of other institutions.

If you're interested in getting involved in forecasting, now's the time. And especially if you're looking for a remote software job at a very fun, mission-driven, and ambitious company.

In particular, we're hiring people to build great experiences for forecasters. In my view this is the greatest unsolved problem in the space: making forecasting natural & engaging.

The link at the top has all of our roles[.](https://apply.workable.com/metaculus/) As of this week we're hiring [Frontend Engineers](https://apply.workable.com/metaculus/j/B36C3B4BF1/) ([Senior Frontend Engineer](https://apply.workable.com/metaculus/j/3EBD0F130C/)) and [Full Stack Engineers](https://apply.workable.com/metaculus/j/409AECAA94/) ([Senior Full Stack Engineer](https://apply.workable.com/metaculus/j/88D35C6884/)). And if you're more interested in AI, we're also still [hiring people directly](https://www.lesswrong.com/posts/JqtPkFqmJtoHTahis/metaculus-is-building-a-team-dedicated-to-ai-forecasting) to study AI risk from a forecasting lens. We're hoping to hire a good number of people very quickly.
# Spectrum of Independence

When I compare our parenting decisions to peers and neighbors' we tend to be near the far end of thinking our kids are ready to do things [on their own](https://www.jefftk.com/p/growing-independence). I don't see other first graders (6y) unaccompanied at the park or [crossing the street](https://www.jefftk.com/p/teaching-street-crossing), and it was only this year that our third grader's (8y) friends started ringing the doorbell to ask if she could go out and play, while she's been going over and ringing theirs for a couple years. Last year when we gave permission for our second grader (7y) and kindergartner (5y) to walk home together the school said yes, but somewhat reluctantly and they didn't remember this situation with a kindergartner before. \[1\]

It's reasonably common for people to point at the past for examples of this level of independence being normal. Reading older children's books the kids wander everywhere, my dad walked to school and back alone in kindergarten, and articles comparing child walksheds by generation are [evergreen viral content](https://www.dailymail.co.uk/news/article-462091/How-children-lost-right-roam-generations.html). The phenomenon of declining children's independence over time is well documented. But what I hadn't internalized until seeing it recently was that even in the US there is still much more variation than I see in my daily life.

A few weeks ago I [brought the kids along](https://www.jefftk.com/p/playing-dances-with-a-kid) to a dance I was playing, and we stayed with a couple who lived adjacent to a farm. They're on good terms with the farmers, and one afternoon they brought us over to look around. While I knew it was common for farm kids to have more independence and responsibility, it was different seeing it in person.

The farmers have several kids, and we ran into their 9yo moving soybeans for storage. They drove past us in a large tractor pulling a short train of wagons, and lined it up carefully with a hopper. They had another tractor set up with an unshrouded [power take-off](https://en.wikipedia.org/wiki/Power_take-off) for an auger to get the beans up into the bin. After transferring the beans in the first trailer, they moved the train up to unload the second. All of this was completely on their own, and they moved like someone who had done this many times.

[![](https://www.jefftk.com/child-farm-responsibility.jpg)](https://www.jefftk.com/child-farm-responsibility-big.jpg)

I'm not saying I'd let my kids operate serious machinery unsupervised at the same age—this is dangerous work even for adults—but it illustrates how wide a range there is in what different people consider appropriate and normal levels of independence and makes me feel less like I have extreme views.

  
\[1\] The school is about 120 years old so they definitely have had many kindergartners walk home before, but apparently not within institutional memory.
# Has anyone increased their AGI timelines?

The question:  Within the last 5-10 years, is there is any person or group that has openly increased their AGI timelines?  
Ideally, they would have at least two different estimates (years apart?), with the most recent estimate showing that they think AGI is further into the future than the prior estimate(s).    
  
Background: Whenever I see posts about AGI timelines, they all seem to be decreasing (or staying the same, with methodological differences making some comparisons difficult). I wondered if I'm missing some subset of people or forecasters that have looked at recent developments and thought that AGI will come later not sooner.  Another framing, am I wrong if I say, "Almost everyone is decreasing their timelines and no one is increasing them" ?
# Apple Cider Syrup

We bought some apple cider when we went apple picking a few weeks ago, and while it's tasty stuff it seems it's not what people wanted to be [spending their sweet budget on](https://www.jefftk.com/p/how-much-is-a-sweet): we still had 2/3 of it left three weeks later. I decided to reduce it and make syrup, partly so that it would keep longer but mostly because apple cider syrup seemed like a fun ingredient to have available.

I put the cider in the slow cooker, the same way Julia [reduces sap to make maple syrup](https://thewholesky.wordpress.com/2021/01/15/you-can-make-maple-syrup-in-a-slow-cooker/). Possibly I should have filtered it first, but I didn't. I left it all day with the lid cracked so liquid could escape. Occasionally one of us would check on it, including tipping the lid over the sink to ditch the condensate, and ~8hr later it was down to ~1/8 the original volume.

[![](https://www.jefftk.com/apple-cider-syrup-jar-spoon.jpg)](https://www.jefftk.com/apple-cider-syrup-jar-spoon-big.jpg)

It's very flavorful, and I can think of a lot of ways I might incorporate it into desserts.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02rqciYAggJDcCmQzJEXSznWif3argUUrAqKeUfnKonpYSuFpXk9n4Va4ka2DUwSxrl)*
# Playground Game: Monster

Here's a game I play with the kids at the park a lot:

*   You play on a play structure.
    
*   One person is the "monster"
    
*   Anyone they tag becomes a monster as well.
    
*   Monsters are not allowed on the horizontal surfaces of the play structure.
    
*   Non-monsters can't go on the ground.
    
*   Little kid monsters (7 and younger?) are "baby monsters" who can still go on horizontal surfaces.
    

Usually I'm the monster over and over. The game typically involves a lot of trying to sneak up on people and reach them through the bars, or climbing around on the railings trying to corner people.

[![](https://www.jefftk.com/monster-on-the-play-structure.jpg)](https://www.jefftk.com/monster-on-the-play-structure-big.jpg)

This works best on a play structure with lots of railings and where the main platform is at about adult shoulder height. You often need to tweak the rules for the specific structure and kids you're playing with, trying to keep it balanced: if games are running much too long/short then shift the rules slightly towards or away from the monster.

When I was little I think we used to call this "don't touch the ground tag", though looking online now that seems to be generally used to refer to a different game.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0maNkWAjrn7oF1zseno4tNgsWpeCG664aYJtw7WQNZrsFwW2kd5U6TwPm5DSpCUjYl)*
# Exams-Only Universities

**Quality**: fast write-up (~45 minutes); little researched

I want a university that only does exams, which would include a description of what you need to know for the exams. Bonus would be suggestions for textbooks and online classes to learn the material, but that's optional.

# Cost
The costs are:
- Exam creation
- Supervision
- Scoring

**Exam creation** is a fixed cost per topic per round (you need to change the exam each year).

**Supervision**, if in person, is partly a fixed cost per location, and partly a variable cost. (Someone can supervise students doing exams on different topics.) Online supervision would also be an option ideally.

**Scoring** is partially a fixed cost (the part that can be automated) and partially a variable cost (the part that need to be reviewed manually).

# Frequency
Frequency of exams could change with demand, but, to start with, you could have one cheap exam per topic per year, timed with the end of normal school years. This would be the exam most people would take, so it would spread the fixed cost among more people. This might also be a more valuable test because it could position you on a normal curve more precisely given the greater amount of people taking it. There could also be more expensive tests throughout the year—more expensive given the fixed cost would be spread among fewer people.

# Problem it solves

What this solves:
1) Decouples learning and exams
a) You can learn at your own pace (whether that's more slowly of faster)
b) You can learn from wherever you want (maybe you want to learn from different places for different topics)

2) Creates standardized tests making it easier to compare the competency of students between different schools

Why wanting exams in the first place? Because many governments want them for immigration and many organizations want them for employees.

# Questions I have
1) Has this been done?

2) Could this be done? Could you have a university that only does exams and emits degrees that are recognized by the US government?

# Current state
There are universities that don't require you to attend classes, but I still find those non-ideal because it's:

1) **Bias**: In my experience, students that attend have an unfair advantage—Not because they learn more, but because teachers literally (and even intentionally) give unfair hints about their exams to reward students for showing up and justifying their salary.

2) **Inconvenient**:

a) You need to be in a specific location for a few years.

b) You can still do some exams only once per year, which prevents doing a degree faster.

# Future

While all I'm asking in this post is for non-zero universities to offer this so that the demand for it gets fulfilled, I also have the impression there would be significant benefits at a societal level from more fully decoupling exams and learning in general. It seems to me like test scores would become more meaningful, and it would become cheaper to get scored.
# Is there any discussion on avoiding being Dutch-booked or otherwise taken advantage of one's bounded rationality by refusing to engage?

From [https://www.gwern.net/mugging:](https://www.gwern.net/mugging:)

> One way to try to escape a mugging is to unilaterally declare that all probabilities below a certain small probability will be treated as zero. With the right pair of lower limit and mugger’s credibility, the mugging will not take place. But such a ad hoc method violates common axioms of probability theory, and thus we can expect there to be repercussions. 
> 
> It turns out to be easy to turn such a person into a money pump, if not by the mugging itself. Suppose your friend adopts this position, and he says specifically that any probabilities less than or equal to 1/20 are 0. You then suggest a game; the two of you will roll a d20 die, and if the die turns up 1-19, you will pay him one penny and if the die turns up 20, he pays you one dollar - no, one bazillion dollars. Your friend then calculates: there is a 19/20 chance that he will win a little money, and there is a 1/20 chance he will lose a lot of money - but wait, 1/20 is too small to matter! It is zero chance, by his rule. So, there is no way he can lose money on this game and he can only make money. 
> 
> He is of course wrong, and on the 5th roll you walk away with everything he owns. (And you can do this as many times as you like.)

Of course, that's not how a sane street-rational person would think! They would not play for "one bazillion dollars" no matter the odds. In general, detecting a sufficiently intelligent adversarial entity tends to result in avoiding the interaction altogether (if you are inured enough to Nigerian princes offering billions in an email). And yet I cannot find any LW discussion on when and if to engage and when to not engage, except in an occasional comment.
# 2022 LessWrong Census?

From 2011-2017, there was an annual LessWrong census/survey. Much like a national census, this provided a valuable lens into the demographics and beliefs of LessWrongers. Unfortunately, this tradition appears to have stopped in recent years, with the exception of a mini-revival in [2020](https://www.lesswrong.com/posts/mf2XKPfT2Brj8mho7/2020-lesswrong-demographics-survey). (Scott Alexander appears to have moved the census to SlateStarCodex.)

From what I've read, this is mainly due to of a lack of will/time among those in the community to run this project, and not a general judgement against the census. 

If this is the case, I'd like to start a new version of the census this year, with a greater emphasis on alignment research/beliefs about AI and timelines. 

Is this a good idea?
# [Book Review] "Station Eleven" by Emily St. John Mandel

# TL;DR

Is it good? Yes.

Should you read it? I don't know.

# Spoilers Ahead

*Station Eleven* isn't the kind of book I'd normally read but I wanted to go to a book club and *Station Eleven* is what they were reading so I bought a copy and quickly read it over the weekend—just in time for the Tuesday MeetUp.

*Station Eleven* is an interweaving of several stories which take place before and after a pandemic[^1] destroys civilization. The stories before the apocalypse follow actor Arthur Leander and his first wife Miranda who is drawing a comic book series *Dr. Eleven*. The stories after the apocalypse follow Kirsten, an actress in a nomadic band of actors and musicians who call themselves the Travelling Symphany. Kirsten likes reading *Dr. Eleven* and old tabloids about Arthur Leander.

[^1]: *Station Eleven* was written before COVID-19.

It wasn't until I was halfway through that book that I started to like it and it wasn't until after finishing the book that I figured out what it was about.

The most important scene in the book is a nighttime party with a bunch of Hollywood bigwigs. Arthur is flirting with his soon-to-be second wife. Miranda is unable to connect with her husband's friends so she goes outside to play with the dog instead.

Miranda has no friends of her own. She does well in business, eventually becoming an executive at a major shipping company. Her sole passion is drawing *Dr. Eleven* which she works on her entire life. She never publishes it commercially. She just does a tiny self-published printing, from which she gives two copies to her by-that-time-ex-husband Arthur. Arthur doesn't really get *Dr. Eleven* so he gives them away.

Arthur has friends but they're all superficial. There's nobody he cares about who cares about him. Even though he's famous.

Kirsten does have friends: all the members of the Travelling Symphany. *Station Eleven* is ostensibly about how everyone was connected in civilization and how that goes away after the apocalypse. But all the personal connections of the characters before the apocalypse were transient and superficial. Their relationships after the apocalypse aren't. The author Emily Mandel doesn't romanticize the apocalypse. Life after the apocalypse is nasty, brutish and short. But it's a nice juxtoposition.

Why is this book good?

The scenes with the Travelling Symphany are exciting adventure, of course.

But what really got to me was Miranda's loneliness. Her closest connection is to Kirsten, a girl she will never meet. The way Miranda felt at that nighttime party is a way I have felt many, many times.
# Google Search as a Washed Up Service Dog: "I HALP!"

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ca4b6ca84d755da9acae575c874226e9bade3a0fa8d6b0dc.png)

As is customary, I whined (on IRC) about Google Search being unhelpful or actively counterproductive with the results these days and how it is not what it used to be. Specifically, I wanted to know something very niche, why many LTE modems sometimes take too long to disconnect. The hits were actively unhelpful, like "why does my internet keep disconnecting?" and there was NOT A SINGLE RELEVANT ENTRY in the first 10 pages. (And yes, I know the information is out there, you just can't form a good query to find it.) Here is the best response by far (emphasis mine):

<quanticle> It reminds me of my friend's dog. This dog was a candidate to be a guide/helper dog for disabled people but, for whatever reason, just couldn't pass the final test of guide dog academy. So, it had like... 90% of the training and instincts of a guide dog, but didn't have anyone to actually guide. **So it'd do some cute-but-in-retrospect-neurotic things like, if it noticed my friend cooking, it'd bring her a screwdriver? Why? Because, look, you're doing a thing, and here's a thing! I HALP**  
<quanticle> **Google is like that dog**
# AI Safety Unconference NeurIPS 2022

The AI Safety Unconference brings together persons interested in aspects of AI safety, from technical AI safety problems to issues of governance of AI. As an unconference, it aims to foster valuable social interactions between participants, through moderated discussion groups, one-on-ones, lightning talks, free-form interactions.

**Date**: Monday November 28 from 9:00 to 16:00, in New Orleans, alongside NeurIPS 2022.

**Location**: Near the Convention Center - exact location to be communicated directly with registered participants.

Participate
-----------

Fill the **[Application form](https://airtable.com/shr5uLL4tkTuHKOQh)**.

The event is private, free, with a maximum of 100 participants.

Join the [chat room on Matrix](https://matrix.to/#/!kTsOmBGiyQWKmETKhS:one.ems.host?via=one.ems.host), to discuss online before or during the event.

Contribute facilitated discussion (30 min) or a lightning talk (10min). We also welcome other ideas of activities - feel free reach out to the organizers about that.

Agenda
------

*   09:00-09:30 - Event opening, breakfast is served
*   09:30-12:00 - Facilitated discussions and 1:1s
*   12:00-13:30 - Lightning talks, lunch is served
*   13:30-15:30 - Facilitated discussions and 1:1s
*   15:30-16:00 - Event closing

Vegan breakfast and lunch are provided, along with all-day drinks and snacks.

The [Swapcard](https://www.swapcard.com/app/swapcard) app is used for scheduling 1:1 meetings with other participants and registering to facilitated discussions.

We have confirmed participants from the following organizations: Mila, Uni. of Stanford, Anthropic, OpenAI, UC Berkeley, Uni. of Toronto, ETH & Max Planck Institute, Uni. of Cambridge, Vector Institute, NYU, ETH Zurich, DeepMind, Oxford, MIT, and more.

Further information
-------------------

**Testimonial of past events** ([2018](https://aisafetyunconference.info/2018), [2019](https://aisafetyunconference.info/2019)):

*   A great way to meet the best people in the area and propel daring ideas forward. — [Stuart Armstrong](https://www.fhi.ox.ac.uk/team/stuart-armstrong/)
*   The event was a great place to meet others with shared research interests. I particularly enjoyed the small discussion groups that exposed me to new perspectives. — [Adam Gleave](https://www.gleave.me/)

**Background readings**:

*   Center for AI Safety's [About AI Risk](https://safe.ai/about-ai-risk)
*   [Krakovna's AI safety resources](https://vkrakovna.wordpress.com/ai-safety-resources)
*   [Alignment newsletter](https://rohinshah.com/alignment-newsletter/)
*   [Superintelligence: Paths, Dangers, Strategies](https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies) by Nick Bostrom

Acknowledgements
----------------

The event is organized in partnership with the [Center for AI Safety](https://safe.ai/).

Organizers:

*   Orpheus Lummis
*   Mauricio H. Luduena

Thanks Nisan Stiennon for funding the event.

For any questions or feedback, reach out to [info@aisafetyevents.org](mailto:info@aisafetyevents.org).
# Thinking About Mastodon

Social networks are normally very sticky—you want to be wherever the people you want to interact with are—so it's hard for alternatives to succeed. With the upheaval around Twitter, however, a lot of people are considering moving away from it, and rethinking social media choices in general, which makes this an unusual opportunity for community migration. I'm especially seeing a lot of suggestions that people move to Mastodon.

Instead of being run by a single company that builds the software and runs the servers (Facebook, Twitter, etc) Mastodon is more like email. You sign up for an account with some provider ("server"), but can talk to people regardless of which provider they're signed up with ("federation"). There's an open protocol ([ActivityPub](https://en.wikipedia.org/wiki/ActivityPub)), open-source server software ([Mastodon](https://github.com/mastodon/mastodon), [Pleroma](https://git.pleroma.social/pleroma/pleroma), etc), and [many servers](https://joinmastodon.org/servers) you could join.

Overall I'm pretty pessimistic about Mastodon, even if we only imagine futures in which lots of people move to it, because of spam. Handling spam on a centralized platform is hard but manageable; federation changes this dramatically for the worse. Imagine you're running a server, and you get an incoming message (comment, like, DM, etc) from an account on another server. Today the default is to let it through, but as Mastodon gets larger there will be more and more money to be made in spamming there, and the larger a fraction of those incoming messages would be spam. Many of the signals centralized platforms use to detect spammers (local activity, account age, IP addresses, etc) are not available in a federated context, leaving server admins (and the software they delegate to) very little information for identifying and blocking spam. It's at least as hard a problem as email spam filtering, probably harder because of shorter messages, and I expect this makes it very hard to run a server that doesn't drown its users in incoming spam and reliably gets its messages out to other servers. Maybe we get to the equivalent of everyone using Gmail/Hotmail/Yahoo and anyone with a different provider has a very frustrating time?

Still, I'm happy to give it a try, and if it does collapse in a pile of spam, oh well. Does anyone have a server they'd recommend? I'm not that excited about the idea of joining a server at random because I'm trusting the server admin not to impersonate me, and also because apparently being on a server with an interesting local timeline is fun.

(I considered self-hosting, but an email server is already more complex than I want to manage and a Mastodon server is [much more so](https://docs.joinmastodon.org/user/run-your-own/).)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid09KyP15ciQr2gzX79pqKseDFHkiToHRXFrNQDwQzJcDwWm9HnQYwFuKDUqrTaEKWYl)*
# Intercept article about lab accidents

The Intercept made FOI requests to gather information about American lab accidents. Additionally, their article discusses the current lack of legislation to require commonsense biosafety strategy from researchers.

> Unless they work with the most dangerous pathogens, biolabs don’t have to register with the U.S. government. As a result, there is little visibility into the biosafety of experiments carried out by private companies or foundations.
> 
> “Your favorite tech billionaire could, with their own money, do basically whatever the hell they want with any pathogen,” said Rocco Casagrande, managing director of Gryphon Scientific, a biosafety advisory firm that has advised NIH on biosafety standards. “They could take the measles virus and intentionally try to make it vaccine-resistant and more pathogenic in their garage. If they’re doing it for legitimate research purposes in their own minds, they can do so wildly, unsafely, and no one can stop them.”
> 
> \[...\]
> 
> “There are some significant holes,” said Filippa Lentzos, an expert on biosecurity and biological threats at King’s College London. Biosafety protocols are “not embedded in statutory law. It’s tied to funding.”
> 
> \[...\]

> Between April 2013 and March 2014, the University of North Carolina at Chapel Hill [reported five mouse escapes](https://www.documentcloud.org/documents/23178920-hvistendahlthe-intercept-unc-03-07-14-incident-report-to-nih-osp), including one of an animal that had been infected with SARS four days earlier. In a letter to NIH, a biosafety specialist argued that the frequency of escapes was due to the “complex research taking place at our institute” rather than a failure of training, noting that several teams at the university use a breed of transgenic mouse known for its unpredictable behavior.
# Binance is buying FTX.com: How did it happen and what are the implications? 

Sources: SBF's [Twitter thread](https://twitter.com/SBF_FTX/status/1590012124864348160), in the [news](https://www.cnbc.com/2022/11/08/binance-offers-to-buy-ftxs-non-us-operations-to-fix-liquidity-crunch.html).

In particular, I'm interested how will this influence SBF's wealth and thus activities such as [Future Fund](https://ftxfuturefund.org/).
# Inverse scaling can become U-shaped

**Edit**: Here's a [great comment by Ethan Perez](https://www.alignmentforum.org/editPost?postId=LvKmjKMvozpdmiQhP&key=7f6ea8c7c618187a899886ef6cc6f8) that caveats this result, that I'd recommend reading for context.

This is a paper by folks at Quoc Le's team at Google that examines the winning tasks from [Round 1 of Anthropic's Inverse Scaling Prize](https://www.alignmentforum.org/posts/iznohbCPFkeB9kAJL/inverse-scaling-prize-round-1-winners). They find that 3/4 of the winning tasks — which exhibited *negative* returns-to-scale when tested on LMs up to the scale of [Gopher](https://storage.googleapis.com/deepmind-media/research/language-research/Training%20Gopher.pdf) (280B) — *go back* to exhibiting *positive* returns-to-scale at even greater model sizes such as [PaLM](https://arxiv.org/abs/2204.02311) (540B).

**The abstract in full:**

> Although scaling language models improves performance on a range of tasks, there are apparently some scenarios where scaling hurts performance. For instance, the Inverse Scaling Prize Round 1 identified four ''inverse scaling'' tasks, for which performance gets worse for larger models. These tasks were evaluated on models of up to 280B parameters, trained up to 500 zettaFLOPs of compute.   
>  
> 
> This paper takes a closer look at these four tasks. We evaluate models of up to 540B parameters, trained on five times more compute than those evaluated in the Inverse Scaling Prize. With this increased range of model sizes and training compute, three out of the four tasks exhibit what we call ''U-shaped scaling'' -- performance decreases up to a certain model size, and then increases again up to the largest model evaluated. One hypothesis is that U-shaped scaling occurs when a task comprises a ''true task'' and a ''distractor task''. Medium-size models can do the distractor task, which hurts performance, while only large-enough models can ignore the distractor task and do the true task. The existence of U-shaped scaling implies that inverse scaling may not hold for larger models. 
> 
>   
> Second, we evaluate the inverse scaling tasks using chain-of-thought (CoT) prompting, in addition to basic prompting without CoT. With CoT prompting, all four tasks show either U-shaped scaling or positive scaling, achieving perfect solve rates on two tasks and several sub-tasks. This suggests that the term "inverse scaling task" is under-specified -- a given task may be inverse scaling for one prompt but positive or U-shaped scaling for a different prompt.

Key figure from the paper is below, showing results for LMs up to [PaLM 540B](https://arxiv.org/abs/2204.02311). Note that positive scaling resumes for 3/4 of the inverse scaling tasks at the 2.5e24 FLOPs datapoint, which indeed [corresponds](https://www.aitracker.org/) exactly to vanilla PaLM 540B.[^1cz6ferw6y3]

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8c91ed2fd5007a35bc1e5931d4bffb543469b80173c6a727.png)

[^1cz6ferw6y3]: From Table 22 in the PaLM paper.
# Trying Mastodon

One of the comments on [yesterday's post](https://www.jefftk.com/p/thinking-about-mastodon) about Mastodon was someone on LW inviting me to join their instance, and I now have an account: [@jefftk@schelling.pt](https://schelling.pt/@jefftk) \[EDIT: moved to [@jefftk@mastodon.mit.edu](https://mastodon.mit.edu/@jefftk) after `schelling.pt` disappeared\]. Right now everything is pretty empty: I don't have a good way to find people I know who are using it and there's no aggressive [People You May Know](https://www.facebook.com/help/336320879782850) like on FB. So: if you're on Mastodon let me know your username and I'll add you.

I'm sure I'll have more thoughts on what it's like to use it later when I've actually done things, but so far:

*   Following someone whose account I did know ([Harris](https://chromamine.com/), [@harris@social.coop](https://social.coop/@harris)) was a little clunky. I was able to get it to work by searching for their username on my server (instead of clicking "follow" on their profile). It also wasn't immediate, so I initially thought it didn't work.
    
*   The local timeline seems uninteresting. Possibly this is being on the wrong server, or possibly this just isn't a feature I'd like.
    
*   The global timeline is even less interesting, which isn't surprising since "posts on Mastodon" is not a good proxy for "has interests and conversational norms I like".
    
*   Posts are limited in length, so instead of text-only versions of my posts with a link, like I do on FB ([example](https://www.facebook.com/jefftk/posts/pfbid09KyP15ciQr2gzX79pqKseDFHkiToHRXFrNQDwQzJcDwWm9HnQYwFuKDUqrTaEKWYl)), I posted a short summary and a link ([example](https://schelling.pt/@jefftk/109309625298242103)).
    
*   Mostly it feels pretty normal, interface-wise, and things seem to work how I would expect.
    

I'm planning to start linking my posts there, at least for a while, and see how it goes.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02acZ8eih7tyukEUxo9UvCNUvsNbU3dHfisE7vG9TZTXSGeeNGkK1QRWDhwHSnLruAl), [mastodon](https://schelling.pt/@jefftk/109309752858449584)*
# Chord Notation

When I [want to](https://www.jefftk.com/p/simple-contra-dance-piano) [write chords](https://www.jefftk.com/p/simple-contra-dance-piano-more-left-hand) I use a quick [chord chart](https://en.wikipedia.org/wiki/Chord_chart) notation. There are lots of different ways of doing this, so I wanted to explain how the particular system I've ended up using works. Here's an example:

|: Am AmG Am  CG  Am AmG FG Am :|
|: Am /   AmC DE  Am /   G  Am :|

Compare to chords written above standard musical notation:

[![](https://www.jefftk.com/AnitaAnderson-BusStopReel-P1--Am-reel.jpg)](https://www.jefftk.com/AnitaAnderson-BusStopReel-P1--Am-reel-big.jpg)

Each letter normally represents two beats. When there's a different chord for each beat you write them squished together: "AmG" is two beats, while "Am G" is four. A slash ("/") means to stay on the previous chord for another two beats. There's an extra space every eight beats and a new line every sixteen, so you can keep track of where you are in the form. I'll also add extra spaces to line things up vertically to again make it clearer how it fits the form. It works best in a monospaced font.

Here's another example:

|: A  AD A E  A  AD AE A :|
|: Bm /  A /  Bm /  AE A :|

[![](https://www.jefftk.com/Trad-ShenandoahFalls-P1--A-reel.jpg)](https://www.jefftk.com/Trad-ShenandoahFalls-P1--A-reel-big.jpg)

Sometimes I'll use [numbers](https://en.wikipedia.org/wiki/Nashville_Number_System) instead of chord names, to make it easier to think about what's happening musically. The previous tune would be:

|: 1  14 1 5  1  14 15 1 :|
|: 2m /  1 /  2m /  15 1 :|

Note that "14" is "1" and "4" on successive beats, not the fourteenth scale degree.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid036sfqho39itSpPAtFGuCET5K5xHyW2y8Jo72BLozTocVuJjq2d3p44MwRJCv1hfyml), [mastodon](https://schelling.pt/@jefftk/109315942859290504)*
# Desiderata for an Adversarial Prior

Based on the discussion in the comments on [https://www.lesswrong.com/posts/6XsZi9aFWdds8bWsy/is-there-any-discussion-on-avoiding-being-dutch-booked-or](https://www.lesswrong.com/posts/6XsZi9aFWdds8bWsy/is-there-any-discussion-on-avoiding-being-dutch-booked-or).

Epistemic status: I am not an expert in the subject matter, so not confident, seems useful to have, I could not find anything discussion online. 

As I understand it, [Pascal Mugging](https://www.lesswrong.com/tag/pascal-s-mugging) can either be an honest opportunity for a large gain in utility (e.g. a Powerball ticket?) if the mugger is honest, or an adversarial attempt to exploit the agent (e.g. a Nigerian Prince [Advance Fee Scam](https://en.wikipedia.org/wiki/Advance-fee_scam)) if not. Most people have some skills in telling the two apart, usually erring on the side of caution. However, it is not clear how to formalize this approach. Most of the discussion focuses on ignoring super-tiny probabilities, and accepting a utility loss penalty, where a proverbial $10 on a busy sidewalk gets ignored because if it were real someone would have picked it up already, a situation humans navigate quite successfully most of the time.

I suspect that Pascal Mugging detection might be better approached from a different direction: instead of starting from "I know a liar when I see one" and the rule of thumb "Most too-good-to-be-true proposals are just that" and trying to formalize it for the case of "tiny probabilities of vast utilities", one could try to start with a more general logic of "adversarial detection", and then apply it to the special case of Pascal Mugging.

How would one go about telling apart an indifferent interaction, where something like the Solomonoff's universal prior is appropriate, from an adversarial one, where it runs into trouble? Whatever the approach, it is likely to have the following properties:

*   In a non-adversarial case it reduces to the universal prior.
*   An attack by a "less-intelligent" adversary is severely penalized, the larger the difference in "intelligence" the harsher. 
*   Conversely, an attack by a "vastly more intelligent" adversary is indistinguishable from the non-adversarial case.

In the Pascal Mugger's case, the really clever one will get your money without you realizing what happened (see  point 3 above), the dumb one will go away empty handed, and there is a whole range in between. There are probably some other desiderata that I am missing here.
# Is full self-driving an AGI-complete problem?

I've felt for quite a while that full self-driving (automated driving without human supervision through arbitrary road systems) is a problem that is deceptively hard. Yes, it is possible to map a route and navigate on a road mesh, do lane following, and even obstacle avoidance using current systems. With LIDAR and well-trained avoidance systems things like Waymo can operate in constrained urban environments.

But as soon as the training wheels are off and the environment becomes unconstrained, the entire problem stops being about just whether we can design an agent which has driving capabilities and becomes "can we make a vehicle which can predict agent-agent dynamics?" If we think about the full range of human road behaviors we must consider adversarial attacks on the system such as:

*   Blocking it from entering a lane
*   Boxing it in and forcing it off the road into obstacles
*   Throwing paint/eggs/rocks at its vision systems
*   Using deceptive tactics (e.g. pretend to be a road worker) to vandalize it and/or steal from its cargo
*   Intentionally standing in its path to delay it
*   Making blind turns in front of it
*   Running into traffic

In addition to agent-agent problems, we must also consider road hazards:

*   Poorly maintained roads with damaging potholes
*   Sinkholes which have disabled the road
*   Eroded road sides with dangerous falls
*   Road debris from land slides
*   Road debris from other vehicles

In these situations, a perfectly rule-following automaton behaves well below human level in preventing delay or damage to itself. Do these scenarios require AGI for a level-5 autonomous vehicle to reach human level? Are the benefits from above-average performance in normal traffic enough to offset the risk of subhuman performance in extrema?
# LessWrong Poll on AGI

This is just a short post to promote this [Pol.is poll](https://pol.is/7yvdmar3nj) to loosely aggregate opinions about AGI / AGI x-risk of the LessWrong community. This poll is less structured than a questionnaire and anyone can add statements that others can agree/disagree with. Pol.is then automatically generates a nice visualization of similar stances.

I recently used Pol.is to gather complaints of PhD students at my department and it exceeded my expectations, so I thought why not try it here. If this is a success I might make a follow-up post with any interesting findings. These things can be slow to get started so be sure to come back to vote on new statements that others have made.

Here is the link: [https://pol.is/7yvdmar3nj](https://pol.is/7yvdmar3nj)