Harley-ml/lesswrong huggingface dataset:
# Bands And Low-stakes Dances

When I got started playing for contra dances in the Boston area there were several area dances you could consider "low stakes". Small dances, couldn't pay very much, less enthusiasm from established bands. And so they were relatively willing to book bands that were starting out and friendly, encouraging, and tolerant of the mistakes of inexperience.

I think this was a big part of why so many solid bands and dance musicians have come out of this area: no requirement that you be good enough to hold your own at a "big" dance to start getting experience playing for dancing as a band. For example, looking back at my calendar, the [Free Raisins](https://www.freeraisins.com/) played eighteen different evenings before we played our first "big" one ([Concord Friday](https://challcontra.weebly.com/)).

Unfortunately this is much less of a thing than it used to be. Several closed pre-pandemic (I miss the MIT contra dance so much!) and others haven't come back (yet?) or have switched to house bands. There's the [BIDA](https://www.bidadance.org/) [open band](https://www.jefftk.com/p/why-does-the-bida-open-band-work-well) and (even lower stakes) family dance band, but while I think this kind of experience is also valuable there's a different kind of learning that comes from playing in a small group, playing sets you've worked up, and being fully responsible for the music.

I don't have a great solution here: it's hard to start something intentionally low-stakes and people generally would prefer to run the kind of event that lots of people want to attend and has great music. But I do think the environment of the early 2010s with many opportunities for new bands to learn to be dance bands was pretty special, and I'd love it if we could bring back something similar, or at least something with similar effects.

(This same dynamic exists even more strongly for "techno contra": the only opportunities to play are major special events, even more high-profile! But I also am less invested in the success of this dance form, so this bothers me less.)
# How do you feel about LessWrong these days? [Open feedback thread]

Hello! This is jacobjacob from the LessWrong / Lightcone team. 

**This is a meta thread for you to share any thoughts, feelings, feedback or other stuff about LessWrong, that's been on your mind. **

Examples of things you might share: 

*   "I really like agree/disagree voting!"
*   "What's up with all this Dialogues stuff? It's confusing... 
*   "Hm... it seems like recently the vibe on the site has changed somehow... in particular \[*insert 10 paragraphs*\]"

...or anything else! 

The point of this thread is to give you an affordance to share anything that's been on your mind, in a place where you know that a team member will be listening. 

(We're a small team and have to prioritise what we work on, so I of course don't promise to action everything mentioned here. But I will at least listen to all of it!)

I haven't seen any public threads like this for a while. Maybe there's a lot of boiling feelings out there about the site that never get voiced? Or maybe y'all don't have more to share than what I find out from just reading normal comments, posts, metrics, and Intercom comments? Well, here's one way to find out! I'm really curious to ask  and see how people feel about the site. 

So, how do you feel about LessWrong these days? Feel free to leave your answers below.
# Metaculus Launches Chinese AI Chips Tournament, Supporting Institute for AI Policy and Strategy Research

Metaculus’s [Chinese AI Chips Tournament](https://www.metaculus.com/tournament/chinese-ai-chips/?has_group=false&project=2784&order_by=-activity) is live: Forecast and support research by the [Institute for AI Policy and Strategy](https://www.iaps.ai/) (IAPS), a think tank that works to reduce risks from the development and deployment of frontier AI systems.

Better understanding China’s ability to produce its own AI chips over the coming years will help shed light on key issues for AI governance, such as:

*   How will Chinese AI systems [compare](https://twitter.com/benmcottier/status/1729902830650003672) to systems developed by frontrunner labs like OpenAI?
*   Will we see major, state-sanctioned AI chip [smuggling](https://www.iaps.ai/research/ai-chip-smuggling-into-china) operations?

[**Learn more**](https://www.metaculus.com/tournament/chinese-ai-chips/?has_group=false&project=2784&order_by=-activity) **about the tournament and start forecasting on these and other questions:**

*   [How much of Nvidia's quarterly 2024 revenue will come from the Chinese market?](https://www.metaculus.com/questions/20164/nvidia-quarterly-revenue-from-china-in-2024/)
*   [Will any Chinese semiconductor foundry have 20% or more global semiconductor market share before 2033?](https://www.metaculus.com/questions/20157/chinese-foundry-with-20-of-semi-market/)
*   [Will a Chinese firm market an EUV photolithography machine before 2033?](https://www.metaculus.com/questions/20135/chinese-euv-machine-marketed-by-2033/)  
     

[*Read a primer on Chinese AI chips*](https://docs.google.com/document/d/1r-aLT67d2OQnuhZJZihkr0KxxB-PsDMKMut2flaYQ2c/edit#heading=h.j3nf69tpx6k9) *by IAPS researcher Erich Grunewald and independent researcher Chris Phenicie.*
# Bucket Brigade: Likely End-of-Life

During the pandemic a group of us wrote [some software](https://echo.jefftk.com/) to allow people to sing and make music together in real time over the internet, by accounting for latency instead of minimizing it. The initial goal was to support group singing at the [2020 Secular Solstice](https://www.jefftk.com/p/secular-solstice-2020), after which I reworked the UI to be a better fit for [singing parties](https://www.jefftk.com/p/bucket-brigade-updates). It doesn't get much use now that it's safer to music together in person, but geographically distributed groups still sing with it a bit.

I'd been hoping to host it indefinitely, and my usual approach is to writing very low upkeep software is to [avoid dependencies](https://www.jefftk.com/p/designing-low-upkeep-software). Unfortunately writing video chat software directly on top of browser APIs is enough work that I took a dependency on Twilio's [Programmable Video](https://www.twilio.com/docs/video/api) product. Which they've now announced they're shutting down.

They've given a year for migration, with a final date of 2024-12-05, but I'm not excited enough about this project to port it to a new video API. If someone else is, let me know, otherwise Bucket Brigade will shut down then as well.
# Google Gemini Announced

Google just announced Gemini, and Hassabis claims that "in each of the 50 different subject areas that we tested it on, it's as good as the best expert humans in those areas"

> State-of-the-art performance

> We've been rigorously testing our Gemini models and evaluating their performance on a wide variety of tasks. From natural image, audio and video understanding to mathematical reasoning, Gemini Ultra’s performance exceeds current state-of-the-art results on 30 of the 32 widely-used academic benchmarks used in large language model (LLM) research and development.

> With a score of 90.0%, Gemini Ultra is the first model to outperform human experts on MMLU (massive multitask language understanding), which uses a combination of 57 subjects such as math, physics, history, law, medicine and ethics for testing both world knowledge and problem-solving abilities.

> Our new benchmark approach to MMLU enables Gemini to use its reasoning capabilities to think more carefully before answering difficult questions, leading to significant improvements over just using its first impression.

It also seems like it can understand video, which is new for multimodal models (GPT-4 cannot do this currently).
# ROSE Ringing in 2024

The Guild has undergone many changes over the last year, and we'd like to start fresh with everyone for 2024. This will be a social hangout that will include a party game, brief discussion about where the Guild is at, and what we plan for the next year. 

We will meet in the Less Wrong Gather Town digital space, which is a virtual hangout space that allows you to move around. After spawning in Gather Town, head north to 'Wentworth Hall'. It's a cozy space with fireplaces and whiteboards.
# Class consciousness for those against the class system


Each person is special. Amanda is in a class of her own, Fred is in a class of his own. Amanda has many different properties: where she came from, what she looks like, what behaviors she habitually does, what she knows and doesn't know, what her plans are, how she speaks, how she would behave if given power, etc. Fred has his own versions of those properties. They overlap partially but not completely with Amanda's properties. 

When conflict arises, sides must be chosen [citation needed]. One side has people with one set of properties, the other side has people with the negation of those properties. Sides can be chosen out loud and explicitly, or in silence and implicitly. 

Sometimes one side is more coordinated with itself than the other side is. For example: business cartels can collude to raise prices. Since customers coordinate with each other less, they don't know that they're being exploited. And the cartels can hack systems, such as the government, to prevent competitors from breaking the cartel. 

In the case of workers vs. capitalists, workers can gain "class consciousness": awareness of themselves as a group of people with shared properties, who are in a conflict with another group of people. 

But what about those who object to class consciousness? Those who wish to resolve conflict based on reason, negotiation, justice, goal-factoring, pulling the rope sideways? 

Such people have a reflexive horror at the inclination towards class consciousness. They view it as degrading, Molochian, and fundamentally suicidal. By choosing to escalate class conflict, you are choosing a bid for sole sovereignty over a pile of ash, and rejecting the hope of shared sovereignty over a beautiful growing world. 

Because they have a reflexive horror at the inclination towards class consciousness, those who are against the class system choose to not see, not acknowledge, not speak about, and not understand the existence of classes, class conflict, and people engaging in class conflict. It is understandable to wish for a world where there is no reality that you thereby ignore, when you make that choice. And it is understandable to reflexively feel that by choosing to not ignore that reality, you make it more real. 

However, as Trotsky said: "You may not be interested in the dialectic, but the dialectic is interested in you." There is a war, and to choose to ignore it is to roll over and die. How can those against the class system gain appropriate class consciousness without being thereby destroyed?
# "Model UN Solutions"

When I was in high school, because I was on the history bowl team the teacher who advised the model UN club recruited me to play as their delegate in various “historical committees” like the Roman Senate or 1789 French Assembly. I never engaged in any normal committees since you couldn’t undertake false flag attacks or convince the Pope to excommunicate other delegates.

In most committees, as far as I can tell, players represent countries trying to pass a resolution addressing some topic like climate change that’s decided beforehand. An award is given to the player the facilitator decides is the “best delegate”—an unwritten combination of speaking ability, social dominance, and accurately representing (or at least not fatally misunderstanding) your assigned country’s positions and interests.

I often make a mental metaphor about “model UN discussions” and “model UN solutions.” Model UN discussions revolve around people expecting to be rewarded for making many remarks, even though their actual positions could be expressed simply or don’t permit much elaboration.

This leads to the “model UN solutions,” which have a few types, e.g.

*   [Applause lights](https://www.lesswrong.com/posts/dLbkrPu5STNCBLRjr/applause-lights): You could just say buzzwords or unobjectionable trivialities (“When addressing the climate change question we should consider the interests of all the relevant stakeholders. We should apply neither an {extreme viewpoint} nor {the opposite extreme}”)
*   **Unspecified solutions**: You could give very little information that uniquely identifies a specific change from the status quo in the listener’s mind. At the extreme you get a lot of remarks of the form “To address the problem we should {devote resources} to {solving the problem}” where the bracketed parts are replaced with phrases that aren’t much more specific (“To address climate change we should set up task forces to identify the best technological and policy approaches”)
*   **Tradeoff-ignorant solutions**: You could even give a directional suggestion but avoid any consideration of the relevant costs or tradeoffs (“We should fund a new educational outreach program related to climate change”).

You could imagine responses that try to identify empty remarks:

*   Ask whether anyone holds the opposite of the remark.
*   Ask how a proposed solution is specifically different from the status quo.
*   Ask who loses out in a proposal (or how resources will be reallocated). Sometimes no one loses out but more often this is just an unstated tradeoff.
# Without - MicroFiction 250 words

This micro fiction is a love letter to my husband, Mati Roy, a most dedicated life logger who would capture my entire life as well, if he could.

**Without**

“I'm out of vanilla. It calls for a teaspoon but we all know that's a minimum.” Distraught, she threw the empty bottle into the trash and began to measure out the flour. 

“I'll go get some,” he said, uncertain of what else to do. He stepped into the hallway and held out his hand. A fresh bottle of vanilla extract appeared in his hand. He had no idea what would upset her or confuse her. She was brand new. 

“Really?” delighted when she saw it, she gazed directly into his eyes. “My hero!” 

His heart shattered and glowed and tumbled in intense mixed emotions as their eyes met and she praised him with her generous smile. 

It was her, and it wasn't. How could he tell? She looked just the same. Her eyes, though. Was she really there? 

She gave him a quick kiss on the cheek and went back to mixing. He could feel the warmth of her, but the smell of her was missing. *Did I do the right thing?*

The ache of missing her had been there all along but it now crashed like a waiting tsunami. *Of course I did. I can't do this without her.*

He sat down on the stool and watched her. 

He'd always planned to bring her back. When the doctor asked him to go somewhere quiet, to sit, he knew. He was ready. 

But now that she was here, in front of him, he realized he wasn't ready at all.
# Deeply Cover Car Crashes?

When there is a train, plane, or bus crash, it's newsworthy: it doesn't happen very often, lots of lives at stake, lots of people are interested in it. Multiple news outlets will send out reporters, and we will hear a lot of details. On the other hand, a car crash does not get this treatment unless there is something unusual about it like a [driverless](https://www.jefftk.com/p/uber-self-driving-crash) [car](https://www.nbcnews.com/tech/tech-news/driver-hits-pedestrian-pushing-path-self-driving-car-san-francisco-rcna118603) or an already [newsworthy](https://www.wbur.org/news/2023/06/07/boston-mayor-michelle-wu-car-crash) [person](https://www.wcvb.com/article/boston-city-councilor-kendra-lara-speeding-jamaica-plain-crash/44523573#) involved.

The effects are not great: while driving is relatively dangerous, both to the occupants and people outside, our sense of danger and impact is poorly calibrated by the news we read. My guess is that most people's intuitive sense of the danger of cars versus trains, planes, and buses has been distorted by this coverage, where most people, say, do not expect buses to be [>16x safer](https://injuryfacts.nsc.org/home-and-community/safety-topics/deaths-by-transportation-mode/) than cars. This also affects our regulatory system, where we push to make high-capacity transportation modes safer ([usually](https://www.jefftk.com/p/make-buses-dangerous)) even when this makes them [less competitive with cars](https://www.jefftk.com/p/in-light-of-crashes-we-should-not-make-buses-more-safe), [shifting more people](https://jdwise.blogspot.com/2012/11/for-safetys-sake.html) to driving, and increasing risk overall.

News outlets deciding that car crashes are sufficiently routine that most are not worth deep coverage is probably a commercially reasonable decision, where they expect stories that wouldn't get enough readers or viewers to justify the time investment. But I wonder whether groups trying to shift people away from cars could make up the difference. They could fund a newspaper to deeply investigate crashes, looking into the circumstances that led to the incident and illustrating the human impact of the tragedy. Just because something happens often doesn't mean each time matters less.

(I'm somewhat nervous about "pay for more coverage of events that point in the direction you advocate". I don't know how much this is currently accepted, or what approaches news outlets have in place to keep this kind of arrangementf from producing distorted coverage?)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02KiWCdPqMnga9125awYckZxU4fSJ5TBQev5UspjDKJckfNsghsSv39Dy3AvtuTUWl), [mastodon](https://mastodon.mit.edu/@jefftk/111558415653526399)*
# Implicitly Typed C

I don't know any good reason to do this, but if you would rather be writing Python or JavaScript here's something you can do with a C compiler:

$ cat tmp.c
foo(x) {
  return x+5;
}
bar() {
  return 4;
}
main() {
  printf("%d\\n", foo(bar()));
}

$ gcc -w -o tmp.out tmp.c && ./tmp.out
9

This code takes advantage of a historical quirk of C where types are assumed to be `int` unless otherwise specified: `foo(x) {...}` is equivalent to `int foo(int x) {...}`. Additionally the `printf` works because gcc includes `stdio.h` by default, and `main` is special-cased to assume a final `return 0`.

I've occasionally used this style when writing [example code](https://www.jefftk.com/p/a-function-that-returns-twice-fork) to remove visual noise, but it's probably not a good idea there either.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02nitahYnUikPzhwFi7hijABLxoJDkcidVGQhWdTjR5stg3QyDVeHA7iWe47dS2NAkl), [mastodon](https://mastodon.mit.edu/@jefftk/111562639856474798)*
# Enhancing intelligence by banging your head on the wall

The [Sudden Savant Syndrome](https://www.brainandlife.org/articles/understanding-creativity-acquired-savant-syndrome/) is a rare phenomenon in which an otherwise normal person gets some kind of brain injury and immediately develops a new skill. The linked article tells the story of a 40-years old guy who banged his head against a wall while swimming, and woke up with a huge talent for playing piano ([relevant video](https://www.youtube.com/watch?v=GTHDuZo7G3Y)). Now, I've spent 15 years in formal music training and I can ensure you that nobody can fake that kind of talent without spending years in actual piano practice.

[Here](https://www.youtube.com/watch?v=qX6ONPQGBfo)'s the story of another guy who banged his head and became ~~a math genius~~ better with math; you can find several other stories like that. And maybe most puzzling of all is [this paper](https://wmjonline.org/wp-content/uploads/2021/120/1/69.pdf), describing a dozen cases of sudden savants who didn't even bang their head, and acquired instant skill while doing nothing in particular.

I vaguely remember one sudden savant story being mentioned on a children book by Terry Deary, presented in his usual "haha, here's a funny trivia" way. But even as a child, I was pretty shocked to read that. Like, seriously? You could become a math genius just by *banging your head on the wall in some very precise way*? The concept lurked in a corner on my mind ever since.

I don't think that Sudden Savant Syndrome is just a scam; there are too many documented cases and most kind of talent are very, very difficult to fake. But if true, why there are so surprisingly few studies on that? Why is no one spending billions of dollars to replicate it in a controlled way? This is a genuine question; I know very little about biology and neuroscience, but it surely sounds way easier than rewriting the genetic code of every neuron in the brain...
# Taboo "procrastination"

There are specific reasons for why you are procrastinating. There are specific solutions to these reasons. Bundling it all up into the "procrastination" black box, or offering solutions to the "procrastination" problem, is a problem.  Specific diagnosis is not always straightforward, but it'll get you farther than "procrastination".

"I'm not working on X because I feel oppressed by a ['should'](https://mindingourway.com/should-considered-harmful/) injunction." 

"I'm not working on X because my hunter-gatherer instincts always attribute more weight to the immediate choice ahead of me that implies the least energy expenditure, and I'll be damned if I'm ever calling that 'laziness'." 

"I'm not working on X because that would be the very first building block of X, and going from 0 to 1 is the hardest thing ever." 

"I'm not working on X because I am dehydrated/sleepy/hungry and that is actually having a noticeable effect and sapping my morale." 

"I'm not working on X because X is a big leap and that scares me. The status quo is nice." 

"I'm not working on X because I'm working on Y, Z, W, and V all at once, and those are short term and X is long term, and so even though X is more important overall, my algorithm gives more weight to the other letters." 

"I'm not working on X because I feel defeatist right now. My P(doom) is one hell of an [ugh field](https://www.lesswrong.com/posts/EFQ3F6kmt4WHXRqik/ugh-fields)." 

"I'm not working on Y because Y is my hobby, and I would feel guilty if I did it instead of X. But then I'm not doing X either, so I'm doing the completely pointless thing in the middle where all I do is scroll." 

"I'm not working on X because everybody told me working on X was useless or socially embarrassing. I carry my social brain everywhere I go, even when I'm working alone. (2 million years of human evolution, baby.)" 

"I'm not working on X because X really, really, doesn't look attractive right now. I know that if I walk back into X it'll be in the same mess I left it in last time, and I'll have to mop it all up." 

If you can think of other specific reasons for why I'm not working on X, I'd love to n+1.
# AI Views Snapshots

(*Cross-posted* [*from Twitter*](https://twitter.com/robbensinger/status/1734699436176298131)*, and therefore optimized somewhat for simplicity.*)

Recent discussions of AI x-risk in places like Twitter tend to focus on "are you in the Rightthink Tribe, or the Wrongthink Tribe?". Are you a doomer? An accelerationist? An EA? A techno-optimist?

I'm pretty sure these discussions would go *way* better if the discussion looked less like that. More concrete claims, details, and probabilities; fewer vague slogans and vague expressions of certainty.

As a start, I made this image (also available [as a Google Drawing](https://docs.google.com/drawings/d/1Vd_3P_EGUb08Ag1z0NvymaFjF2_XiOlF3TPU76WB8bg/edit)):

![Image](https://pbs.twimg.com/media/GBL_QlraoAA-hlv?format=jpg&name=large)

(**Added:** [Web version](https://ai-views-snapshots.tetratopia.foundation/) made by Tetraspace.)

I obviously left out lots of other important and interesting questions, but I think this is OK as a conversation-starter. I've encouraged Twitter regulars to share their own versions of this image, or similar images, as a nucleus for conversation (and a way to directly clarify what people's actual views are, beyond the stereotypes and slogans).

If you want to see a filled-out example, here's mine (though you may not want to look if you prefer to give answers that are less anchored): [Google Drawing link](https://docs.google.com/drawings/d/16K6V1BQ4UPI4xKpoEag50ofEEdoSkL0wMu3kPzjtOX4/edit).
# Suggestions for net positive LLM research

I am starting a PhD in computer science, focusing on agent foundations so far, which is great. I intend to continue devoting at least half my time to agent foundations.

However, for several reasons, it seems to be important for me to do some applied work, particularly with LLMs:

1.  I believe I'm otherwise pretty well positioned to get an impactful job at Google DeepMind, but apparently some impressive machine learning engineering creds are necessary to get on even the safety team currently.
2.  My PhD supervisor is pushing for me to work on LLMs, though he seems to be pretty flexible about the details.
3.  As much fun as math is, I also value developing my practical skills (in fact, half the fun of learning things is becoming more powerful in the real world).
4.  LLM experts seem likely to be in high demand right now, though I am not sure how long that will last.

Now, I've spent the last couple of years mostly studying AIXI and its foundations. I'm pretty comfortable with standard deep learning algorithms and libraries and I have some industry experience with machine learning engineering, but I am not an expert on NLP, LLMs, or prosiac alignment. Therefore, I am looking for suggestions from the community about LLM related research projects that would satisfy as many as possible of the following criteria:

1.  Is not directly focused on improving frontier model capabilities (for ethical reasons; though my timelines seem to be longer than the average lesswronger, I'm not able to accept the risk that I am wrong).
2.  Produces mundane utility. I find it much more fulfilling to work on things that I can see becoming useful to people, and I also want a measure of my success which is as concrete as possible. 
3.  Contributes to prosiac alignment. It would be particularly nice if the experimental/engineering work involved is likely to inform my ideas for mathematical alignment research.
4.  Machine learning engineering/research creds.

Any suggestions are appreciated. I may also link this question to a Manifold market in the future (probably "conditional on working full time at DeepMind within 18 months of graduation, which areas of research did my PhD thesis include") or something along those lines. Thanks!
# Red Line Ashmont Train is Now Approaching

The MBTA has now fully fixed the sign issue I was [complaining about](https://www.jefftk.com/p/attention-passengers-not-for-signs)! It's great:

[![Red Line
Ashmont Train is Now Approaching](https://www.jefftk.com/red-line-ashmont-train-is-now-approaching.jpg)](https://www.jefftk.com/red-line-ashmont-train-is-now-approaching-big.jpg)

For years I grumbled inside whenever I was approaching a station and saw the digital signs switch to "Attention passengers, the next red line train". This could signal any of:

*   25%: My train is coming, I can catch it if I hustle.
*   25%: I'm missing my train.
*   50%: A train I don't care about is doing something.

If I waited the sign would switch to, for example, "to Ashmont is now approaching", but time when I'm trying to decide whether to hurry to catch a train is unusually precious!

The issue was that they were displaying a transcription of the audio announcement instead of tailoring the textual notification to the medium. Several months ago they fixed the [arriving](https://www.jefftk.com/p/alewife-train-is-now-arriving) version, and recently they fixed the "approaching" version as well. Yay!

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid09N48FLuZteEWNdB4nLx1vUTXwuamiDHHmQKwUCriup1j1NBPFV3XyPe944BZvfUxl)*
# Update on Chinese IQ-related gene panels

It turns out that Chinese 23-and-me-esque gene panel already include intelligence markers! For example, 23mofang uses Interleukin 3 as a proxy for  brain volume, citing [this paper](https://sci-hub.ru/10.1371/journal.pone.0050375). It's... only significant in women. Not a good sign. But the cited study notes a Danish gene-correlation study that included brain scans, from which they obtained brain volume. Apparently, the correlation is true across races.

In any case, I've been reaching out to past coworkers, and they agree with my assessment that a polygenic database for the purpose of embryo selection would be easy and mostly cheap to do. Being Chinese, we of course have no issues with including factors like eye color, height, intelligence, ect. However, I have several questions before I proceed further.

1.  What is the overall demand? How many customers would be interested? What specific traits are parents most interested in? Funding is the single biggest obstacle I face, and I will be applying for AstralCodexTen funding. If enough interest is displayed or if I get charitable funding, my gut feeling is that I can offer analysis for <$100 per genome, maybe even <$10. 
2.  Would Westerners accept Chinese IQ/educational attainment metrics? The easiest metric to use would be GaoKao scores, but would that be legible to Westerners? Would ratings of attended college be acceptable? What about the applicability of the results across different races?
3.  Publications/computer code. Would I need to publish a paper to be considered legitimate?
4.  Reputational concerns. What Western institutions should I be expected to be blacklisted from because of this?

I have previously published genome-wide association studies regarding cancer outcomes, so I can actually do the analysis and write-up myself. I just need the raw data, which I should be able to obtain by paying a sequencing company with a gene bank to call back their past clients and ask for GaoKao scores.

I would really appreciate it if anyone can direct me towards a source of European ancestry genomic data with labeled donor information (education, height, ect).

Update: For some reason, I didn't consider using previous research like the [n=1.1 million educational attainment study](https://www.nature.com/articles/s41588-018-0147-3). It seems... I can just do everything by myself? I'll sleep on it, but it is certainly interesting.
# Talking With People Who Speak to Congressional Staffers about AI risk

A conversation with [Jason Green-Lowe and Jakub Kraus](https://www.aipolicy.us/) of the Center for AI Policy. They've met with 50+ congressional staffers in DC about AI regulation efforts and are in the process of drafting a model bill. This is a somewhat entry-level conversation, good for getting an idea of what's going on over there without getting very wonky.
# Where Does Adversarial Pressure Come From?

You could see on LW a dialogue like this:

> AI Pessimist: Look, you should apply security mindset when thinking about aligning superintelligence. If you have superintelligence, all your safety measures are under enormous optimization pressure, which will seek how to crack them.
> 
> Skeptic: I agree that if we have misaligned superintelligence, it will game whatever specification we give it and kill us or worse. But why would we have misaligned superintelligence in the first place? **Where does adversarial pressure come from?**

And this is a legitimate question! I always feel the statement "optimization pressure causes weird things" to be intuitive, but many people don't share that attitude and it's not as crisp as I've wanted.

Let's begin with the question: what does it mean for your security assumption to be broken? It means that your security assumption happens to be false. Security assumptions are not locks or doors; you can't break them through physical effort. If all your security assumptions are true (including "this set of assumptions is sufficient for safety"), you are safe. You are only in danger if your security assumptions are false.

If so, why do we have to deal with pressure on these assumptions? Because you are running systems whose very first task is to learn what is true and what is false.

That's it. We don't need anything else to justify necessity to treat overall situation as adversarily-optimized. It's an implementation-agnostic statement. It doesn't matter whether your system is agentic or non-agentic, symbolic or connectionist - if you increase the ability of your system to figure out things as high as possible, your system is going to shoot you in the leg if your security assumptions are false.

Old models of adversarial pressure presented something like this: the system has a plan-generator and plan-evaluator. If your evaluator is not maximally robust, your generator is going to find a plan which satisfies the evaluator but is actually bad. This construction seems unnatural: why would we want to build a generator-of-all-possible-plans? Why wouldn't we want to build a generator-of-similar-to-past-plans, at least?

But we can reframe the situation in other settings. Let's take the alignment of a DL system. Our security assumptions can be, for example:

1.  Our training data contains only Good Things.
2.  Our learning algorithm can learn only Good Things.
3.  Learned Good Things are going to overcome Bad Things.
4.  If Bad Things make their way inside the system, our validation strategy will catch that.

Suppose you deleted a Bad Thing from the training data. But what if you leave in the data a weird Bad-Thing-shaped hole, which can be a sufficient hint for a sufficiently smart learning system?

At this moment, in case when your security assumption is false, you find yourself in situation when you would want your AI to be less powerful, i.e., in adversarial situation. Therefore, you really need to apply security mindset.
# Pope Francis shares thoughts on responsible AI development

Main sections (taken from the article):

1. The progress of science and technology as a path to peace
2. The future of artificial intelligence: between promise and risk
3. The technology of the future: machines that “learn” by themselves
4. The sense of limit in the technocratic paradigm
5. Burning issues for ethics
6. Shall we turn swords into ploughshares?
7. Challenges for education
8. Challenges for the development of international law
# Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

Links: [Blog](https://openai.com/research/weak-to-strong-generalization), [Paper](https://cdn.openai.com/papers/weak-to-strong-generalization.pdf).

**Abstract**:

> Widely used alignment techniques, such as reinforcement learning from human feedback (RLHF), rely on the ability of humans to supervise model behavior—for example, to evaluate whether a model faithfully followed instructions or generated safe outputs. However, future superhuman models will behave in complex ways too difficult for humans to reliably evaluate; humans will only be able to weakly supervise superhuman models. We study an analogy to this problem: can weak model supervision elicit the full capabilities of a much stronger model? We test this using a range of pretrained language models in the GPT-4 family on natural language processing (NLP), chess, and reward modeling tasks. We find that when we naively finetune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors, a phenomenon we call weak-to-strong generalization. However, we are still far from recovering the full capabilities of strong models with naive finetuning alone, suggesting that techniques like RLHF may scale poorly to superhuman models without further work. We find that simple methods can often significantly improve weak-to-strong generalization: for example, when finetuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence loss, we can recover close to GPT-3.5-level performance on NLP tasks. Our results suggest that it is feasible to make empirical progress today on a fundamental challenge of aligning superhuman models.
# The Serendipity of Density

One thing I really appreciate about living in Somerville is how density facilitates serendipity. Higher density means more of the places we might want to go are within walking distance, and then walking (and high density again) means we're more likely to run into friends.

This afternoon we were walking Lily (9y) to a sleepover and we passed one of the nearby playgrounds. Lily and Anna (7y) saw some school friends and ran ahead to say hi. Anna wanted to stay and play, and I asked one of the parents if they'd be up for having Anna while I finished the dropoff. They were happy to (we take each other's kids a decent amount) and Anna got to have a much more fun half hour. Then, after I got back, we hung out at the playground with friends for a while longer before heading home for dinner.

This certainly would have been possible to arrange intentionally, with a bunch of communication, but if we had been driving I expect Anna wouldn't have ended up spending this time with her friends. I would guess that the majority of times we go out we run into someone, though it's much more common that we stop and hang out together than leave one of the kids and split up.

This sort of thing wasn't a common experience at all in the West Medford neighborhood I grew up in. It's an area not far from here, but probably only a third the density: the lots are about twice the size and there are more single-family houses. Distances meant we would usually drive, and even if our family had decided to primarily walk you don't run into friends when out walking unless your friends also tend to be out on foot. Another contribution is also that our kids' school is very close and draws mostly from the immediate neighborhood, which means their friends are almost all within easy walking distance. Whereas my elementary school drew from all over Medford and I only had one school friend whose house I could walk to, and only for two of the six years.

Americans often worry that increasing density would lead to their neighborhoods becoming impersonal. As someone who lives in one of the densest municipalities in the country, I think this is backwards: proximity fosters community.
# Reviving a 2015 MacBook

A few days ago I broke the screen on my MacBook: I unplugged the power adapter and closed the top as usual, but this time the adapter had sprung back. With ~10x leverage \[1\] I cracked the screen, without even noticing.

[![](https://www.jefftk.com/cracked-laptop-screen.jpg)](https://www.jefftk.com/cracked-laptop-screen-big.jpg)

It's completely blank now, except for the ghostly patterns of the backlight. I'm sending it in to be fixed, but need something to work with for the 3-5d they estimate the repair will take.

We had an [early 2015 MacBook](https://support.apple.com/kb/SP715?locale=en_US&viewlocale=en_US) which we'd used for the kids Zoom lessons during the pandemic. I tried using it and it was very sluggish and wouldn't run recent browsers or display most websites. I gave up on it, but came back to it the next day after noticing that it should be able to run [MacOS 12](https://en.wikipedia.org/wiki/MacOS_Monterey) (Monterey, 2021).

Apple's [documentation](https://support.apple.com/en-us/102662) claims that SoftwareUpdate should show compatible versions, but this was running [MacOS 10.10](https://en.wikipedia.org/wiki/OS_X_Yosemite) (Yosemite, 2014) which is old enough that I'm guessing it's no longer compatible with the update servers.

The documentation offered a backup option of downloading MacOS from the App Store, but again because this was so out of date the store wouldn't load. But if offered a backup backup option of downloading a disk image for the installer, up through [MacOS 10.12](https://en.wikipedia.org/wiki/MacOS_Sierra) (Sierra, 2016). Apple's website wouldn't load on this computer, because of certificate issues, but I downloaded it on a different computer. AirDrop didn't work, I think because it was too old, but I got it over on an SD card.

Upgrading took a little while, and then once it was on 10.12 I could follow the backup (AppStore) instructions to upgrade to version 12. Then I let it install updates, which too another while. It was a few hours overall, probably 25min from me and 4hr of waiting for it to churn through the various updates.

I'm writing this from the revived computer, and I'm pretty happy with it! It's from before the sad butterfly keyboard and touchbar era, and while it doesn't feel quite as snappy as my normal computer it's fine. I'm not planning on doing any bioinformatics locally next week, though, where I suspect it would struggle a lot.

This is, unfortunately, a short-lived revival: the hardware doesn't support the two most recent versions of MacOS and Apple will likely drop support for this version in [Fall 2024](https://endoflife.date/macos). But it should get me through the next week, which is the main thing I need from it right now!

  
\[1\] I measure the distance from the hinge to the edge of the screen as 9.5", and to the crack as ~1".
# Talk: "AI Would Be A Lot Less Alarming If We Understood Agents"

This is a linkpost for a talk I gave this past summer for the ALIFE conference. If you haven't heard of it before, ALIFE (short for "artificial life") is a subfield of biology which... well, here are some of the session titles from day 1 of the conference to give the gist:

*   Cellular Automata, Self-Reproduction and Complexity
*   Evolving Robot Bodies and Brains in Unity
*   Self-Organizing Systems with Machine Learning
*   Untangling Cognition: How Information Theory can Demystify Brains

... so you can see how this sort of crowd might be interested in AI alignment.

[Rory Greig](https://www.lesswrong.com/users/rorygreig) and [Simon McGregor](https://www.sussex.ac.uk/profiles/142771/research) definitely saw how such a crowd might be interested in AI alignment, so they organized an alignment workshop at the conference.

I gave this talk as part of that workshop. The stated goal of the talk was to "nerd-snipe ALIFE researchers into working on alignment-relevant questions of agency". It's pretty short (~20 minutes), and aims for a general energy of "hey here's some cool research hooks".

If you want to nerd-snipe technical researchers into thinking about alignment-relevant questions of agency, this talk is a short and relatively fun one to share.

Thankyou to Rory and Simon for organizing, and thankyou to Rory for getting the video posted publicly.
# Assessment of AI safety agendas: think about the downside risk

*This is a post-response to* [*The 'Neglected Approaches' Approach: AE Studio's Alignment Agenda*](https://www.lesswrong.com/posts/qAdDzcBuDBLexb4fC/the-neglected-approaches-approach-ae-studio-s-alignment)*.*

* * *

You evidently follow a variant of [80000hours' framework for comparing (solving) particular problems in terms of expected impact](https://80000hours.org/articles/problem-framework/): **Neglectedness x Scale (potential upside) x Solvability**.

I think for assessing AI safety ideas, agendas, and problems to solve, we should augment the assessment with another factor: the potential for a [Waluigi turn](https://www.lesswrong.com/tag/waluigi-effect), or more prosaically, **the uncertainty about the** ***sign*** **of the impact (scale) and, therefore, the risks of** ***solving*** **the given problem or advancing far on the given agenda**.

This reminds me of Taleb's mantra that to survive, we need to make many bets, but also limit the downside potential of each bet, i.e., the "ruin potential". See "[The Logic of Risk Taking](https://medium.com/incerto/the-logic-of-risk-taking-107bf41029d3)".

Of the approaches that you listed, some sound risky to me in this respect. Particularly, "4. ‘Reinforcement Learning from Neural Feedback’ (RLNF)" -- sounds like a direct invitation for [wireheading](https://www.lesswrong.com/tag/wireheading) to me. More generally, [**scaling BCI in any form and not falling into a dystopia at some stage is akin to walking a tightrope**](https://www.lesswrong.com/posts/d74pb97TAqNKwJkc5/bcis-and-the-ecosystem-of-modular-minds?commentId=EYyie4uXPHuu4gC6F) (at least at the current stage of civilisational maturity, I would say) This speaks to agendas #2 and #3 on your list.

There are also similar qualms about **AI interpretability**: there are at least four posts on LW warning of the potential risks of interpretability:

*   [Why and When Interpretability Work is Dangerous](https://www.lesswrong.com/posts/uhMRgEXabYbWeLc6T/why-and-when-interpretability-work-is-dangerous)
*   [Against Almost Every Theory of Impact of Interpretability](https://www.lesswrong.com/posts/LNA8mubrByG7SFacm/against-almost-every-theory-of-impact-of-interpretability-1)
*   [AGI-Automated Interpretability is Suicide](https://www.lesswrong.com/posts/pQqoTTAnEePRDmZN4/agi-automated-interpretability-is-suicide)
*   [AI interpretability could be harmful?](https://www.lesswrong.com/posts/CRrkKAafopCmhJEBt/ai-interpretability-could-be-harmful)

This speaks to the agenda "9. Neuroscience x mechanistic interpretability" on your list.

### Related earlier posts

*   [Some background for reasoning about dual-use alignment research](https://www.lesswrong.com/posts/zswuToWK6zpYSwmCn/some-background-for-reasoning-about-dual-use-alignment)
*   [Thoughts on the OpenAI alignment plan: will AI research assistants be net-positive for AI existential risk?](https://www.lesswrong.com/posts/6RC3BNopCtzKaTeR6/thoughts-on-the-openai-alignment-plan-will-ai-research)
# Interview: Applications w/ Alice Rigg

Alice Rigg, a mechanistic interpretability researcher from Ottawa, Canada, joins me to discuss her path, and we both reflect on the applications process for research/mentorship programs (*e.g.,* MATS, ASTRA Fellowship, ARENA, AISC).

She also is one of the moderators for a [Mech Interp Discord server](https://discord.gg/KNxJwaytTG) which hosts weekly reading groups (led by Alice).

As a reiteration of my [previous statements](https://www.lesswrong.com/posts/ozDWnEChJwuB5L5wg/documenting-journey-into-ai-safety), I want the Into AI Safety podcast to be a resource for individuals who are interested in getting involved, but are having a difficult time taking the next steps. If you have any advice, feedback, or ideas that you think could help in that endeavor, please reach out!

* * *

In addition to the [Into AI Safety](https://into-ai-safety.github.io/) podcast website, you can find the content on (I believe) all podcast listening platforms (*e.g.,* [Spotify](https://open.spotify.com/show/5AGzrA4jo6mgZuibVabTLM), [Apple Podcasts](https://podcasts.apple.com/us/podcast/into-ai-safety/id1720206246), [Pocket Casts](https://pca.st/t5or5lh5)). If you think that I missed a platform, please let me know.
# What are the best Siderea posts?

I think the [Asshole Filters](https://www.lesswrong.com/posts/n86TNJS8k2Shd84kn/the-asshole-filter) post is underrated and everyone should read and share it. I also like the [post](https://siderea.dreamwidth.org/1237182.html) about economic class vs social class. Do the posts only get worse from there? What are the top 5 or so?