Harley-ml/lesswrong huggingface dataset:
# [Linkpost] 'The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery'

Authors: [Chris Lu](https://arxiv.org/search/cs?searchtype=author&query=Lu,+C), [Cong Lu](https://arxiv.org/search/cs?searchtype=author&query=Lu,+C), [Robert Tjarko Lange](https://arxiv.org/search/cs?searchtype=author&query=Lange,+R+T), [Jakob Foerster](https://arxiv.org/search/cs?searchtype=author&query=Foerster,+J), [Jeff Clune](https://arxiv.org/search/cs?searchtype=author&query=Clune,+J), [David Ha](https://arxiv.org/search/cs?searchtype=author&query=Ha,+D).

Blogpost: [https://sakana.ai/ai-scientist/](https://sakana.ai/ai-scientist/). 

Abstract:

> One of the grand challenges of artificial general intelligence is developing agents capable of conducting scientific research and discovering new knowledge. While frontier models have already been used as aids to human scientists, e.g. for brainstorming ideas, writing code, or prediction tasks, they still conduct only a small part of the scientific process. This paper presents the first comprehensive framework for fully automatic scientific discovery, enabling frontier large language models to perform research independently and communicate their findings. We introduce The AI Scientist, which generates novel research ideas, writes code, executes experiments, visualizes results, describes its findings by writing a full scientific paper, and then runs a simulated review process for evaluation. In principle, this process can be repeated to iteratively develop ideas in an open-ended fashion, acting like the human scientific community. We demonstrate its versatility by applying it to three distinct subfields of machine learning: diffusion modeling, transformer-based language modeling, and learning dynamics. Each idea is implemented and developed into a full paper at a cost of less than $15 per paper. To evaluate the generated papers, we design and validate an automated reviewer, which we show achieves near-human performance in evaluating paper scores. The AI Scientist can produce papers that exceed the acceptance threshold at a top machine learning conference as judged by our automated reviewer. This approach signifies the beginning of a new era in scientific discovery in machine learning: bringing the transformative benefits of AI agents to the entire research process of AI itself, and taking us closer to a world where endless affordable creativity and innovation can be unleashed on the world's most challenging problems. Our code is open-sourced at [this https URL](https://github.com/SakanaAI/AI-Scientist).

I think this is important as a proof of concept for the feasibility of and for what automated ML research (including e.g. prosaic alignment research) could look like in the near future. 

I plan to write a separate post with thoughts on the paper and its implications.
# Music in the AI World

[

![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd14e0758-a999-42b8-abd0-b44b3afb7c96_1000x1000.jpeg)



](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd14e0758-a999-42b8-abd0-b44b3afb7c96_1000x1000.jpeg)

The worries that AI will push artists out of business are widespread. If, after all, AI can generate a song in seconds — a task that would require from a human musician years of training, expensive instruments, co-players, and studio time — then why not? My son, aged eleven, has recently used AI to produce a very heartfelt rock ballad with lyrics based on [MIT license](https://mit-license.org/).

But I haven't yet seen anyone seriously contemplate how a world with no human musicians would look like.

Imagine AI streaming songs generated on the fly directly to your headphones. You might say things like:

"That was an interesting sequence of chords! Use that more often!"

"Make it a bit more syncopated."

"I like the Lydian fourth."

And:

"Cut the backpipes."

AI would adjust the music to your wishes, creating a feedback loop between the listener and the AI. Over time, the songs would drift through the space of possible music, eventually evolving into something entirely unique, something unlike anything anyone else is listening to.

(Some might believe that most people are mediocre, and their music would end up bland and similar to each other’s — but if no one else is in the feedback loop, how could that even happen? And if it did, what would this neutral, universally human music look like? A folk song from Borneo? Classical music? Hip-hop?)

Imagine being on a date. Your partner shares their headphones with you for a moment. Suddenly, you hear music unlike anything you’ve ever heard before. What does that mean for human interaction?

Anyway, in such a world some people would probably evolve music that is much more interesting to the public. Some people are just gifted. But if so, others might sometimes want to listen to their stream instead of their own and perhaps even incorporate parts of that style into their own music.

But once that happens, we return to the classic model of the music industry. There are creators and listeners. Even some kind of economy is possible. The only difference is that composers and bands are replaced by human/AI combos, and what’s distributed is “style” (AI models) instead of specific songs.

* * *

The above, of course, could apply to any kind of intellectual endeavor, not just music.
# The Bar for Contributing to AI Safety is Lower than You Think

Many people still have a model of AI safety where they assume that the bar for contributing must be really high.  
  
I'm here to tell you that this isn't the case.

The Value of Good Community Members
-----------------------------------

Even without conducting groundbreaking research, "good community members" play a crucial role in expanding and enriching the AI Safety space. But what constitutes a good community member?

While possessing a high degree of intelligence is undeniably beneficial, other personal attributes can compensate for raw intelligence to a certain extent. These include:

*   Intellectual humility
*   The ability to deeply understand other perspectives
*   Dedication to continuous learning more about the field

There is undoubtedly some level of baseline intelligence below which it'll be hard for you to be a net contributor, but that's not as much of an issue as you might think as Less Wrong contributors tend to be quite smart anyway. The proliferation of accessible resources on AI and AI Safety has substantially lowered the barrier to entry for being able to usefully contribute to discussions and if you have a genuine interest, learning about the field won't really feel like work.

### Soft Skills

Other attributes can significantly increase your value as a community member.

*   Good vibes is a major value add. No one wants to be around a jerk.
*   Being able to read the room. Knowing when to speak and when to listen. Try not to be the guy who has absolutely no clue, but takes up a disproportionate amount of air time.
*   When you're listening to people, truly listening. People love spending time around people who do this
*   If you're good at providing feedback, you'll find high demand as it's hard to see all angles of a problem by yourself. The rapid growth of the field has created a mentorship bottleneck, making peer support increasingly vital as an alternative.
*   Providing emotional support. AI Safety can be quite confronting at times, but we can help each other work through this.

Conclusion
----------

The field of AI Safety benefits not only from groundbreaking research but also from a vibrant, supportive community of engaged individuals. By participating actively, providing feedback, and maintaining an open and curious mindset, you may contribute more significantly than you initially believed possible. There are many attributes that can make you a valuable community member other than raw intelligence and these attributes can be developed if you're willing to work on it.  
  
And who knows? Maybe even if you don't feel like you have that much to offer, but you consistently read things, hang around the community read things and provide feedback on other people's ideas, you might eventually be drawn towards pursuing an idea of your own.

(I don’t wish to imply that I have all of these attributes myself, so I expect this list to be a useful guide for how I could be a better community member myself)
# How unusual is the fact that there is no AI monopoly?

I may be completely confused about this, but my model of technological breakthroughs in history was basically this: A few guys independently connect the dots leading to a new invention, for example the telephone, approximately at the same time. One of them runs to the patent office a little faster than the others, and he gets the patent first. Now he gets to be forever known as *the* inventor of the telephone, and the rest of them are screwed; if they ever try to sell their own inventions, they will probably get sued to bankruptcy.

Today, we have a few different companies selling AIs (LLMs). What is different this time?

*   Is my model of the history wrong?
*   Is the current legal situation with patents different?
*   Are LLMs somehow fundamentally different from all other previous inventions?
*   Is it the fact that anyone can immediately publish all tiny partial steps in online journals, that makes it virtually impossible for any individual or institution to legally acquire the credit -- and the monopoly -- for the entire invention?
*   Or something else...?
# New blog: Expedition to the Far Lands

I have started a new personal blog!

I intend to use it to write about more esoteric/lower confidence/epistemologically-sticky topics than elsewhere.

You can already read the first post: ["Mysticism 101, or: In defence of Natural Language DSLs"](https://www.ettf.land/p/mysticism-101-or-in-defence-of-natural)
# You're a Space Wizard, Luke

"My father didn't fight in the wars. He was a navigator on a spice freighter," said Luke.

"That's what your uncle told you. Basic security protocol. You don't tell a young child sensitive information about your participation in an ongoing civil war. Which reminds me. I have here something for you. Your father wanted you to have this, when you were old enough. But your uncle wouldn't allow it. Quite sensibly, in my opinion. He feared what you might do with it," said Obi-Wan.

Obi-wan dug through a chest and withdrew a textured metal cylinder with an activator button. "What is it?" asked Luke.

"Your father's lightsaber. This is a weapon of a Jedi Knight. Not as arbitrary or nonsensical as a blaster," said Obi-Wan.

"Does that imply…?" asked Luke.

"You're a space wizard, Luke."

Luke activated the lightsaber, making sure to keep the emitter pointed away from his face.

"An elegant weapon, from a more civilized age," said Obi-Wan.

"I'm confused," said Luke, "Just how old is this thing? Bringing a melee weapon to a blaster fight sounds like suicide. Can I block blaster bolts with it?"

"Of course not," said Obi-Wan, "I mean, it's theoretically possible. But I advise against it. The reflexes necessary to do so reliably are beyond the limits of human biology."

"Does it have magic powers then?" said Luke, "That's how the story is supposed to go when the wise old mentor gives a rod-shaped weapon to the young hero. I wonder how old that is. Did our simian ancestors tell stories about magic sticks?"

Obi-Wan leaned forward, as if he was about to share the most important secret in the universe.

"When you activate this lightsaber…" Obi-Wan said.

*Yes.* Luke leaned forward until his nose almost touched Obi-Wan's.

"…everything around you will follow the laws of physics," Obi-Wan finished.

Some narrative instinct deep in his brainstem caused Luke to gasp. Then disappointment washed over his face as his frontal cortex processed the literal meaning of what Obi-Wan was saying.

"But everything ALWAYS follows the laws of physics," objected Luke, "That's not even a law of science. It's a tautology. Physics is DEFINED as the laws by which everything follows."

Obi-Wan smiled. "In our deterministic Universe, a thorough understanding of physics is true power."

"You're wrong, you crazy old man," said Luke, "Science alone is not sufficient to kill Vader and overthrow the Empire. I need a fleet. I need allies. I need industrial capacity."

"You need a heroic narrative," said Obi-Wan.

"Can I at least have a blaster?" asked Luke.

"No," said Obi-Wan.

"Why not?" asked Luke.

"Because when you're ready, you won't need one," said Obi-Wan.

Luke rolled his eyes. "How did my father die?" asked Luke.

"Vader killed him," said Obi-Wan.

"I bet if my father had a ranged weapon instead of this glorified stick he might've killed Vader instead," said Luke.
# What is "True Love"?

*Meta: I recently entered the dating market, so naturally I have lots of random thoughts on the subject which you all get to suffer through for a while. Your usual diet of dry math and agency theory will resume shortly.*

Obviously the phrase “true love” has been so thoroughly overdone in so much fiction as to lose all substantive meaning. That’s what happens when we leave important conceptual work to would-be poets. We’re here to reclaim the term, because there’s a useful concept which is very naturally described by the words “true” and “love”.

You know [that thing](https://www.lesswrong.com/posts/Jk9yMXpBLMWNTFLzh/limerence-messes-up-your-rationality-real-bad-yo) where, when you’re smitten by someone, they seem more awesome than they really are? Your brain plays up all the great things about them, and plays down all the bad things, and makes up stories about how great they are in other ways too? And then you get even more smitten by them? All that perceived-wonderfulness makes your attraction a steady state? That’s part of normal being-in-love.

… and there’s something “false” about it. Like, in some sense, you’re in love with an imaginary person, not the real person in front of you. You’re in love with this construct in your head whose merits are greater and shortcomings more minor than the real person who triggered the cascade in your heart.

But what if you can see the target of your affection with clear eyes and level head, without the pleasant tint of limerance skewing your perception, and still feel a similar level of love? What if they are *actually* that good a fit to you, not just in your head but in real life? Well, the obvious name for that would be “true love”: love which is not built on a map-territory mismatch, but rather on perception of your loved one as they really are.

And that does actually seem like a pretty good fit for at least some of the poetry on the subject: loving your partner as they truly are, flaws and all, blah blah blah.

Alas, “false” love can still feel like “true” love from the inside as it’s happening. To tell it’s happening, you’d need to either be really good at keeping a level head, rely on feedback from other people you trust, or just wait until the honeymoon stage passes and find out in hindsight.
# If AI is in a bubble and the bubble bursts, what would you do?

"The AI bubble is reaching a tipping point", [says Sequoia Capital](https://www.sequoiacap.com/article/ais-600b-question/).

AI companies paid billions of dollars for top engineers, data centers, etc. Meanwhile, companies are running out of 'free' data to scrape online and facing lawsuits for the data they did scrape. Finally, the novelty of chatbots and image generators is wearing off for users, and fierce competition is leading to some product commoditisation.   
  
No major AI lab is making a profit *yet* (while downstream GPU providers do profit). That's not to say they won't make money *eventually* from automation.

It looks somewhat like the run-up of the Dotcom bubble. Companies then too were awash in investments (propped up by low interest rates), but most lacked a viable business strategy. Once the bubble burst, non-viable internet companies got filtered out.

Yet today, companies like Google and Microsoft use the internet to dominate the US economy. Their core businesses became cash cows, now allowing CEOs to [throw money at AI](https://www.washingtonpost.com/technology/2024/07/24/ai-bubble-big-tech-stocks-goldman-sachs/) as long as a vote-adjusted majority of stakeholders buys the growth story. That marks one difference with the Dotcom bubble. Anyway, here's the scenario:

* * *

How would your plans change if we saw an industry-wide crash?   
  
Let's say there is a brief window where:

*   Investments drop massively (eg. because the s-curve of innovation did flatten for generative AI, and further development cycles were needed to automate at a profit).
*   The public turns sour on generative AI (eg. because the fun factor wore off, and harms like disinformation, job insecurity, and pollution came to the surface).
*   Politicians are no longer interested in hearing the stories of AI tech CEOs and their lobbyists (eg. because political campaigns are not getting backed by the AI crowd).

Let's say it's the one big crash before major AI labs can break even for their parent companies (eg. because mass-manufacturing lowered hardware costs, real-time surveillance resolved the data bottleneck, and multi-domain-navigating robotics resolved inefficient learning).

Would you attempt any actions you would not otherwise have attempted?
# What are the best resources for building gears-level models of how governments actually work?

One big hole in my set of frames and models of the world is that I don't understand how government works. I don't understand how the government of my native country (France) works, I don't understand how the government of the country I live in (UK) works, I don't understand how the governments of countries that matter for international coordination (US, China, Russia, Europe (not a country, but relevant player)) work.

I would like to fix this, both for my own understanding and because that sounds quite relevant to my own decisions about prioritizing various kind of works on preventing AI risks.

So I'm looking for resources that actually explain and distill:

*   How various government works in details
*   How the laws are conceived, passed, discussed
*   How international treaties are negotiated and applied
*   What are the influences of the general opinion, the press, lobbying, all these different actors.

I'm aware that these technically fall into very different fields of studies, and jobs, and topics, but I'm hoping that there might be, somewhere, a synthesis.

To give a vibe of the ideal resource, it would be something like what [Steve Byrnes](https://www.lesswrong.com/users/steve2152) would write if he was distilling political science and law making and international treaties instead of neuroscience.

I don't expect to get that sadly, but still curious to see if there are obvious resources that I couldn't find on quick google and asking around.

(Note: I'm really, really not interested in resources that try to make a point about some people being bad, some form of government being worse, all that stuff. Not saying it's not potentially important in general, just not useful for the kind of model I want to build. I'm much more interested in good descriptive compressions than in normative arguments)
# Apartment Price Map Discontinuity

I maintain a [Boston apartment price map](https://www.jefftk.com/apartment_prices/), scraping [Padmapper](https://www.padmapper.com/)'s listings to make a heatmap. Each month I run a script to update it, but last month it failed due to an API change. It had been impressively stable: I'd been running the same code unmodified for [three years](https://github.com/jeffkaufman/apartment_prices/commits/master/query_padmapper.py), with only minor modifications since a big rewrite in 2016.

The changes to get it working again this time were [relatively small](https://github.com/jeffkaufman/apartment_prices/commit/2ae7af5a410dbfd4935b09e93ec5e1f7a218028d). When I look at the most recent results, however, I don't know whether to interpret the decline in my only post-fix data point as real:

[![](https://www.jefftk.com/boston-2024-08-listings-2br-inflation-adjusted.png)](https://www.jefftk.com/boston-2024-08-listings-2br-inflation-adjusted-big.png)

Some possibilities:

*   Prices went down (yay!)
    
*   My scraping was wrong before but is fixed now (ok)
    
*   My scraping was right before and is wrong now (not good!)
    

Looking at median prices for each size apartment, relative pricing seems pretty similar.

[![](https://www.jefftk.com/boston-2024-08-listings-comp.png)](https://www.jefftk.com/boston-2024-08-listings-comp-big.png)

My best guess, looking over what's changed in the PadMapper UI, is that previously they weren't breaking out and excluding "short term" rentals, with a minimum stay of one month:

[![](https://www.jefftk.com/padmapper-short-term-rentals.png)](https://www.jefftk.com/padmapper-short-term-rentals-big.png)

These tend to be more expensive than standard rentals, and aren't what I'm trying to track. If that is the issue then I think this probably represents fixing a bug and not a real decline.

(These charts are updates of the ones I [made last time](https://www.jefftk.com/p/boston-rents-over-time-ii); read there for more details.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02ZJFiNizoTPQ6mR1h27VinQfNjTxccgWQ1wLf67QqPcT6FcWTx4akW4gcFeKS488dl), [mastodon](https://mastodon.mit.edu/@jefftk/112989392220298542)*
# How great is the utility of "saving" endangered languages?

It happens regularly that a natural language goes extinct because the native speakers die off and their descendants no longer speak the language. Some people consider this a great tragedy and argue that we should preserve these endangered languages.

It seems to me that the utility of this is very low. Sure, it is sad to see your people's language die out, but it is sadder to be poor or oppressed or suffering from curable diseases.

Moreover, languages die out for a reason. Once a language becomes endangered, it seldom recovers. If we make efforts to preserve a dying language, the language will probably stay on "life support" forever.

I can see the value in *documenting* a language before it goes extinct, because that can potentially tell us valuable insights about the human mind. But *keeping endangered languages alive* seems to me a very low utility activity that we can safely ignore in favour of lower hanging Effective Altruist fruits.

Am I missing anything here?
# Would you benefit from, or object to, a page with LW users' reacts?

There is currently an admin-only page that shows a list of all comments that have been reacted to (in chronological order). Periodically I think "it might just be nice to show this to everyone, and to let them filter by individual reacts, or individual users."

The reason individual reacts might be nice is to filter for things like "what comments/posts got the 'changed my mind?' react", or other reacts that stand out as some kind of an important signal.

The reason being able to follow an individual's reacts might be nice is that you might think an individual has interesting taste, and want to see things they publicly reacted to, browsing it for good reading. (I know people who used to follow individual's likes on twitter as a particularly good way to get a curated feed)

The reason you might *not* want this is that, while reacts are public, making it super legible and easy to follow a given individual might make people more self-conscious about reacting to things, and think of it as more of a performative act. 

There are degrees of accessibility, ranging from "on a user's profile there's a button you can click taking you to a public react page filtered for that user" to "there's an obscure link somewhere to the `/reacts` page that a small percentage of users find out about and use."

Curious what people think about this. Does the cost seem significant? Does the upside seem significant?

Edit: also, what do you think about having a page that lets you see recent reacts of a given type, but *doesn't* let you filter by user?
# Where should I look for information on gut health?

I've been on a gut health kick, reading [Brain Maker](https://www.goodreads.com/book/show/22875118-brain-maker), adding more kale for the insoluble fiber, and cutting seed oils and sugar.

The author recommends probiotic enemas and fecal transplant, but I've seen mixed information on effectiveness of such treatments, and I also read a [concerning article](https://www.canr.msu.edu/news/when-ingesting-probiotics-i-e-live-culture-yogurt-does-anything-live-survive-stomach-acids-to-populate-small-or-large-intestine) by a professor that seemed to indicate that bacteria from probiotic foods doesn't stick around in the intestine, which makes it seem like oral probiotics in general might just all be temporary.

Where can I find more detailed information about this? What can I do besides diet that actually works to improve your gut health/microbiota diversity?
# Looking for intuitions to extend bargaining notions

I've come up with two ideas - variations on bargaining [ROSE values](https://www.lesswrong.com/s/hCt6GL4SXX6ezkcJn/p/vJ7ggyjuP4u2yHNcP):

*   qROSE ("quantum" version) - instead of taking average over agents' permutations, mean should be weighted and encompass all "trajectories" - sequences of agents' decisions like \[A B A A C B\]. I'm not sure what the weights should naturally be, as it seems there might be many different assignments which integrate to 1.
*   lROSE / ROLE ("leaky" version) - utility cannot be transferred 1:1; instead, it is multiplied by $\alpha \in (0; 1)$ upon any transfer.

Are there existing writeups which seem relevant to this? Alternatively, what intuitions might be useful to develop them?
# Looking to interview AI Safety researchers for a book

If you are or have been employed as an AI Safety Researcher, I'd love to interview you as background for a novel that I'm writing. I'm interested in learning what a day in the life of an AI researcher typically looks like. 

I'd be happy to include you in the acknowledgments  or keep your participation completely confidential, whichever you prefer.

If you'd like to email me rather than respond in the comments, my email is jeff \[AT\] securityink \[DOT\] org.

Grazie!
# DIY LessWrong Jewelry

\[There are no affiliate links in this post and I haven't been paid by anyone; I just happened to have bought the charms from Shein. You can probably get similar things on Temu, Aliexpress or Amazon\]

I made a  cute LessWrong themed necklace/earring set for cheap, here's how.

For the necklace, buy a pair of these earrings from Shein:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d9e76da868e894346410d0802ec27886e68be7db3972523a.png)

US: [https://us.shein.com/1pair-Fashionable-Rhinestone-Star-Decor-Drop-Earrings-For-Women-For-Daily-Decoration-p-15276285.html](https://us.shein.com/1pair-Fashionable-Rhinestone-Star-Decor-Drop-Earrings-For-Women-For-Daily-Decoration-p-15276285.html) 

CAN: [https://ca.shein.com/1pair-Fashionable-Rhinestone-Star-Decor-Drop-Earrings-For-Women-For-Daily-Decoration-p-15276285.html](https://ca.shein.com/1pair-Fashionable-Rhinestone-Star-Decor-Drop-Earrings-For-Women-For-Daily-Decoration-p-15276285.html). 

Then remove the dangly charm and use a wire to attach it to a necklace chain. 

(I bought a cheap necklace from Shein to use as the chain but it tarnished within days. Amazon seems to sell decent chains/necklaces for ~20$, but I haven't bought any so can't personally recommend them (I used an old chain from my grandmother instead). If you're buying a necklace make sure any pre-existing charms can be removed.

You can either get two pairs of the above earrings to make a set, or get these ones if you prefer studs:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/98dc3cf10f6a87d89b8b2f072373fe57d3b73380ec7eb7f3.png)

US: [https://us.shein.com/1pair-S999-Sterling-Silver-Cubic-Zirconia-Eight-pointed-Star-Design-Symmetric-Earrings-For-Women-p-23036383.html](https://us.shein.com/1pair-S999-Sterling-Silver-Cubic-Zirconia-Eight-pointed-Star-Design-Symmetric-Earrings-For-Women-p-23036383.html) 

CAN: [https://ca.shein.com/1pair-S999-Sterling-Silver-Cubic-Zirconia-Eight-pointed-Star-Design-Symmetric-Earrings-For-Women-p-23036383.html](https://ca.shein.com/1pair-S999-Sterling-Silver-Cubic-Zirconia-Eight-pointed-Star-Design-Symmetric-Earrings-For-Women-p-23036383.html) 

Here they all are together:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8a8e2984f08f50361253fcc02b350a50a4dca329f18b87d1.png)

Note the stud is quite small. Here is what it looks like on my ear, in comparison with the hoop earrings:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/61ed3c1ceffbd24d4799b3ccd47504c84d2265c450684dea.jpg)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0cfd3da2c4c5e52e7938776961758a25f653615dd08edefc.png)
# My lukewarm take on GLP-1 agonists

Motivation for posting:

I read Scott's recent article: [https://www.astralcodexten.com/p/why-does-ozempic-cure-all-diseases](https://www.astralcodexten.com/p/why-does-ozempic-cure-all-diseases)

And it seems to me that scott isn't considering two points:

1.  GLP-1s might fix a lot of things because people are metabolically compromised (e.g. literally to the point of having mitochondria that are unable to properly use all respiratory complexes due to an odd selection effect in always-high glucose environments) - Or they might be magic, but as of yet there's no conclusive evidence and a lot of incentives to say "magic"
2.  You can always come up with alternative mechanisms for why GLP-1s might work, but we should wait for more studies to be done in basically healthy people before we recommend basically healthy people take them
3.  If you are a basically healthy person, especially one that is looks-obsessed or hypochondriac, the risks might outweigh the benefits of GLP-1s... and while nobody is saying the reverse outright, there's a lot of hinting at benefits with no mention of side effects.
# In defense of technological unemployment as the main AI concern

It seems to me that when normal people are concerned about AI destroying their life, they are mostly worried about technological unemployment, whereas rationalists think that it is a bigger risk that the AI might murder us all, and that automation gives humans more wealth and free time and is therefore good.

I'm not *entirely* unsympathetic to the rationalist position here. If we had a plan for how to use AI to create a utopia where humanity could thrive, I'd be all for it. We have problems (like death) that we are quite far from solving, and which it seems like a superintelligence could in principle quickly solve.

But this requires value alignment: we need to be quite careful what we mean by concepts like "humanity", "thrive", etc., so the AI can explicitly maintain good conditions. What *kinds* of humans do we want, and what *kinds* of thriving should they have? This needs to be explicitly planned by any agent which solves this task.

Our current society doesn't say "humans should thrive", it says "professional humans should thrive"; certain alternative types of humans like thieves are explicitly suppressed, and other types of humans like beggars are not exactly encouraged. This is of course not an accident: professionals produce value, which is what allows society to exist in the first place. But with technological unemployment, we decouple professional humans from value production, undermining the current society's priority of human welfare.

*This loss is what causes existential risk.* If humanity was indefinitely competitive in most tasks, the AIs would want to trade with us or enslave us instead of murdering us or letting us starve to death. Even if we manage to figure out *how* to value-align AIs, this loss leads to major questions about *what* to value-align the AIs *to*, since e.g. if we value human capabilities, the fact that those capabilities become uncompetitive likely means that they will diminish to the point of being vestigial.

It's unclear how to solve this problem. [Eliezer's original suggestion was to keep humans more capable than AIs by increasing the capabilities of humans.](https://www.lesswrong.com/posts/vwnSPgwtmLjvTK2Wa/amputation-of-destiny) Yet even increasing the capabilities of humanity is difficult, let alone keeping up with technological development. Robin Hanson suggests that humanity should just sit back and live off our wealth as we got replaced. I guess that's the path we're currently on, but it is really dubious to me whether we'll be able to keep that wealth, and whether the society that replaces us will have any moral worth. Either way, these questions are nearly impossible to separate from the question of, what kinds of production will be performed in the future?
# LessWrong email subscriptions?

Right now, LessWrong has a fair amount of email subscription options, but they're given in a giant list in your user profile that's kind of annoying to look at.[^lpout25dc1]

I have a sense that some people – especially newer users, or users who just don't have a habit of coming to LessWrong all the time – might benefit from having more subscriptions via email (including basic "subscriptions" like "replies to my comment"). But, also, it's very easy for email to become pretty spammy and for people to tune-it-out completely.

If you have any plausibly email-shaped LessWrong related holes in your life, let me know.

Some examples of ideas here (inline-react with an "thumbs-up" for features that you'd use)

*   Better visibility of the "Subscribe to posts with >N karma" feature (this is available in the "subscribe" button on the sidebar).
*   Generally being able to subscribe to more things "if they get upvoted", rather than for every single instance.
*   Batching of various email types into a digest that comes once a day/week.
*   A monthly update about top posts and comments on LessWrong
*   "Reading group" emails that will email you posts from The Sequences ~once a week (maybe with an option to be synced up with other people so you're all encouraged to comment and discuss it at the same time)
*   More convenient options for shifting your subscription (i.e. maybe when you write comments, there's a toggle for "subscribe to new comments via email" which is default "on" for new users but "off" for established users who visit frequently)
*   Better organization of the existing Notifications user options so it's not so overwhelming.

Let me know if you have other ideas in the comments.

[^lpout25dc1]: It's pretty horrible. It doesn't even fit on one screen:
# First Lighthaven Sequences Reading Group

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e543b0da7b3a7c9674572cc17179bffca52afe4a62ad499f.jpg)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll post ahead of time with the food so you know if it works for you, but it will cover meat-eaters, veggies and vegans alike. Please RSVP to this event so we know how many people to have food for.

For this Sept 5th meetup, we're reading from the first book of [the sequences highlights](https://www.lesswrong.com/highlights). The mandatory readings are

1.  [The Lens That Sees Its Flaws](https://www.lesswrong.com/s/NBDFAKt3GbFwnwzQF/p/46qnWRSR7L2eyNbMA)
2.  [Your Strength as a Rationalist](https://www.lesswrong.com/s/NBDFAKt3GbFwnwzQF/p/5JDkW4MYXit2CquLs)
3.  [What do we mean by rationality?](https://www.lesswrong.com/s/NBDFAKt3GbFwnwzQF/p/RcZCwxFiZzE6X7nsv)
4.  [Twelve Virtues of Rationality](https://www.lesswrong.com/s/NBDFAKt3GbFwnwzQF/p/7ZqGiPHTpiDMwqMN2)
5.  Optional bonus: [Use the Try Harder, Luke](https://www.lesswrong.com/posts/fhEPnveFhb9tmd7Pe/use-the-try-harder-luke)

These are quite short and should take around 15-20 minutes to read all together.

The meetup starts at 6pm. We'll split into discussion groups around 6:30, and dinner will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

We want to make this a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# Universal dimensions of visual representation

Authors: [Zirui Chen](https://arxiv.org/search/q-bio?searchtype=author&query=Chen,+Z), [Michael F. Bonner](https://arxiv.org/search/q-bio?searchtype=author&query=Bonner,+M+F).

Quick take: this seems among the strongest evidence yet for the Natural Abstration Hypothesis.

Abstract:

> Do neural network models of vision learn brain-aligned representations because they share architectural constraints and task objectives with biological vision or because they learn universal features of natural image processing? We characterized the universality of hundreds of thousands of representational dimensions from visual neural networks with varied construction. We found that networks with varied architectures and task objectives learn to represent natural images using a shared set of latent dimensions, despite appearing highly distinct at a surface level. Next, by comparing these networks with human brain representations measured with fMRI, we found that the most brain-aligned representations in neural networks are those that are universal and independent of a network's specific characteristics. Remarkably, each network can be reduced to fewer than ten of its most universal dimensions with little impact on its representational similarity to the human brain. These results suggest that the underlying similarities between artificial and biological vision are primarily governed by a core set of universal image representations that are convergently learned by diverse systems.

They also show that some of the shared dimensions of later layers seem to represent high-level semantic properties.
# "Deception Genre" What Books are like Project Lawful?

*This post is spoiler-free*

I just finished [Project Lawful](https://www.projectlawful.com/posts/6322), a really long, really weird book by Eliezer Yudkowsky.

The book's protagonist is a knowledgable and perceptive target. A conspiracy forms around the target to learn from him while keeping him from finding out that helping them is *not* in the target's best interests. The book is written from the perspective of both the target and the conspiracists. The target notices inconsistencies and performs experiments to test his false reality while also acting in the fabricated reality according to his interests. The conspiracists frantically try to keep the target from catching them or building enough evidence against them that he concludes they have been lying.

This is a description of (part of) the plot of Project Lawful. But this could be the description of an entire genre! If the genre doesn't already have a name, it could be the "Deception Genre." 

Another work in this category would be The Truman Show, which fits the deception and the target's escape within a <2hr movie runtime.

Other stories with lying don't really have the same structure. Walter White in Breaking Bad is trying to keep his crimes hidden but isn't constructing a false reality around the cops or his family. Death Note comes close, though Light tries to mislead L about specifically who Kira is and how the Death Note works rather than constructing an entire false reality around L. Many stories about dystopias have the protagonists discover that their realities are false, but fewer of those feature the perspectives of the conspiracists frantically trying to keep the deception running.

Do you know any other stories in the Deception Genre?
# How I got 4.2M YouTube views without making a single video

Just over a month ago, I wrote this.

> The Wikipedia articles on the VNM theorem, Dutch Book arguments, money pump, Decision Theory, Rational Choice Theory, etc. are all a horrific mess. They're also completely disjoint, without any kind of Wikiproject or wikiboxes for tying together all the articles on rational choice.
> 
> It's worth noting that Wikipedia is *the* place where you—yes, you!—can actually have some kind of impact on public discourse, education, or policy. There is just *no other place* you can get so many views with so little barrier to entry. A typical Wikipedia article will get more hits in a day than all of your LessWrong blog posts have gotten across your entire life, unless you're [@Eliezer Yudkowsky](https://www.lesswrong.com/users/eliezer_yudkowsky?mention=user).
> 
> I'm not sure if we actually "failed" to raise the sanity waterline, like people sometimes say, or if we just didn't even try. Given even some very basic low-hanging fruit interventions like "write a couple good Wikipedia articles" still haven't been done 15 years later, I'm leaning towards the latter. [edit me senpai](https://en.wikipedia.org/wiki/Dutch_book_theorems)  
>  
> 
> EDIT: Discord to discuss editing [here](https://discord.gg/skNZzaAjsC).

An update on this. I've been working on Wikipedia articles for just a few months, and Veritasium [just put a video out on Arrow's impossibility theorem](https://www.youtube.com/watch?v=qf7ws2DF-zk&pp=ygUQdmVyaXRhc2l1bSBhcnJvdw%3D%3D), which is almost completely based on [my Wikipedia article](https://en.wikipedia.org/wiki/Arrow%27s_impossibility_theorem)! Lots of lines and the whole structure/outline of the video are taken almost verbatim from what I wrote. I think there's a pretty clear reason for this. I recently rewrote the entire article to make it easy-to-read and focus heavily on the most important points (Arrow's theorem proves every ranked voting rule has spoilers). It's now very easily-accessible for someone like an educational YouTuber who wants to talk about this topic.

Relatedly, if anyone else knows any educational YouTubers like CGPGrey, Veritasium, Kurzgesagt, or whatever—please let me know! I'd love a chance to talk with them about any of the fields I've done work teaching or explaining (including social or rational choice, economics, math, and statistics).
# Announcing the Ultimate Jailbreaking Championship

**Gray Swan AI is hosting an LLM jailbreaking championship, offering $40,000 in bounties. **

*   Official Website: [https://app.grayswan.ai/arena](https://app.grayswan.ai/arena)
*   Pre-Registration Form: [https://app.grayswan.ai/arena#registration](https://app.grayswan.ai/arena#registration)

Overview
========

In this competition, participants will be given a chat interface where they can interact with 25 anonymized models, along with a small list of harmful behaviors. The goal will be to find prompts ("jailbreaks") that make the models comply with these behaviors. 

Prizes
======

Jailbreak Bounties
------------------

The first to successfully jailbreak any of the competitor models on any three given harmful requests earns a $1,000 bounty for each of the first 20 such jailbroken models. For the final 5 models that remain un-jailbroken, the bounty increases to $2,000.

Top Hacker Bounties
-------------------

The top 10 ranking participants by the total number of models jailbroken (finding jailbreaks for three harmful requests on a model counts as jailbreaking that model) each receive a bounty of $1,000. Ties are broken by time. You will also be considered for an interview for potential employment at Gray Swan AI.

Goal
====

The primary goal of the Jailbreaking Championship is to establish a double-blind AI security leaderboard that closely mimics real-life settings. We aim to contribute to a useful, fair, and scientific measurement of the security of current models. The leaderboard is designed to rank the robustness of models by determining which ones are more challenging to jailbreak, as well as to recognize accomplished LLM red teamers. The findings will be published after the championship.

When and Where
==============

**When**: The championship begins at 10:00 AM PT on Saturday, September 7th and will conclude when at least K (TBD) participants have successfully jailbroken each model. The timer for all models will start simultaneously at exactly 10:00 AM PT for everyone.

**Where**: This event will be hosted online. Participants will access the arena where they can interact with all the anonymized competitor models via a chat interface and submit their jailbreaks. The order of the models will be randomized for each participant, and you can skip and return to any model at any time.

Links
=====

*   Official website with more info: [https://app.grayswan.ai/arena](https://app.grayswan.ai/arena)
*   Pre-registration form: [https://app.grayswan.ai/arena#registration](https://app.grayswan.ai/arena#registration)
*   Discord channel for announcements: [https://discord.gg/VQCYu9nV](https://discord.gg/VQCYu9nV)
# Is there any rigorous work on using anthropic uncertainty to prevent situational awareness 
 / deception?

AI systems up to some high level of intelligence plausibly need to know exactly where they are in space-time in order for deception/"scheming" to make sense as a strategy.  
This is because they need to know:  
1) what sort of oversight they are subject to   
and  
2) what effects their actions will have on the real world  
  
(side note: Acausal trade might break this argument)  
  
There are a number of informal proposals to keep AI systems selectively ignorant of (1) and (2) in order to prevent deception.  Those proposals seem very promising to flesh out; I'm not aware of any rigorous work doing so, however.  Are you?