Harley-ml/lesswrong huggingface dataset:
# [Linkpost] Silver Bulletin: For most people, politics is about fitting in

Nate Silver tries to answer the question: "How do people formulate their political beliefs?"

An important epistemological question that is, he says, under-discussed.

He lays out his theory:

> I think political beliefs are primarily formulated by two major forces:
> 
> **Politics as self-interest.** Some issues have legible, material stakes. Rich people have an interest in lower taxes. Sexually active women (and men!) who don’t want to bear children have an interest in easier access to abortion. Members of historically disadvantaged groups have an interest in laws that protect their rights
> 
> **Politics as personal identity — whose team are you on.** But other issues have primarily *symbolic* stakes. These serve as vehicles for individual and group expression — not so much “identity politics” but *politics as identity*. People are trying to figure out where they fit in — who’s on their side and who isn’t. And this works in both directions: people can be attracted to a group or negatively polarized by it. People have [different reasons for arguing about politics](https://www.natesilver.net/p/the-5-types-of-people-who-argue-on), and can derive value from a sense of social belonging and receiving reinforcement that their choices are honorable and righteous.[^1^](https://www.natesilver.net/p/for-most-people-politics-is-about?utm_source=post-email-title&publication_id=1198116&post_id=144184034&utm_campaign=email-post-title&isFreemail=true&r=f8zbh&triedRedirect=true&utm_medium=email#footnote-1-144184034)
> 
> Notice what’s missing from my list? The notion of politics as a *battle of ideas*. This is not to suggest that people don’t hold reasonable moral intuitions about political affairs. But when it comes to mass popular opinion, the number of people who are interested in *ideas for ideas sake* is vanishingly small.
# Can stealth aircraft be detected optically?

5th generation military aircraft are extremely optimised to reduce their radar cross section. It is this ability above all others that makes the f-35 and the f-22 so capable - modern anti aircraft weapons are very good, so the only safe way to fly over a well defended area is not to be seen.

But wouldn't it be fairly trivial to detect a stealth aircraft optically?

This is what an f-35 looks like from underneath at about 10 by 10 pixels:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/88548554741d3606a1040773f7f370f7f75c2f166d775f90.png)

You and I can easily tell what that is (take a step back, or squint). So can GPT4:

> The image shows a silhouette of a fighter jet in the sky, likely flying at high speed. The clear blue sky provides a sharp contrast, making the aircraft's dark outline prominent. The jet's design appears sleek and streamlined, with distinct wings and a tail fin, typical of modern military aircraft.

A specialised image classifier should also be able to do so no problem at all, and much faster.

The patriot radar has a range of about 100km. Let's say we wanted to match that.

An f-35s wingspan is 11m. I want to detect the F-35 at 100kms out in all directions. Then we need one pixel per metre, over the surface of a semisphere with radius 100km.

That equates to a bit less than one hundred billion pixels. You can buy phones with 100 million pixel cameras for under $1000, and the majority of the cost is not the camera. This suggests you could use an array to cover the entire sky for under a million dollars, an order of magnitude less than the cost of a radar. The rest of the infrastructure you would need would also cost, including some beefy GPUs, but not enough to increase the cost more than an order of magnitude.

You'd need two of these to perform triangulation and calculate the distance and size of the object, as well as to exclude false positives (small bird close to the camera). You'd probably want 3 for greater reliability. But that should be relatively simple to do. 

This only works in good weather, but denying the enemy use of their aircraft in good weather seems pretty useful, especially in the middle east where it's continuously sunny half the year.

Also, unlike a radar, this doesn't advertise it's location whenever it's used.  
  
So why isn't this done?  
  
My best guesses are that it would be fairly easy to camouflage the f-35, or that the performance characteristics of this solution wouldn't be good enough to guide missiles, only to detect the existence of the aircraft.
# Which skincare products are evidence-based?

The beauty industry offers a large variety of skincare products (marketed mostly at women), differing both in alleged function and (substantially) in price. However, it's pretty hard to test for yourself how much any of these product help. The feedback loop for things like "getting less wrinkles" is very long.

So, which of these products are actually useful and which are mostly a waste of money? Are more expensive products actually better or just have better branding? How can I find out?

I would guess that sunscreen is definitely helpful, and using *some* moisturizers for face and body is probably helpful. But, what about night cream? Eye cream? So-called "anti-aging"? Exfoliants?
# Why is AGI/ASI Inevitable?

Hello! My name is Amy.  
  
This is my first LessWrong post. I'm about somewhat certain it will be deleted, but I'm giving it a shot anyway, because I've seen this argument thrown around a few places and I still don't understand. I've read a few chunks of the Sequences, and the fundamentals of rationality sequences. 

**What makes artificial general intelligence 'inevitable'? What makes artificial superintelligence 'inevitable'? Can't people decide simply not to build AGI/ASI?**

I'm very, very new to this whole scene, and while I'm personally convinced AGI/ASI is coming, I haven't really been convinced it's inevitable, the way so many people online (mostly Twitter!) seem convinced. 

While I'd appreciate to hear your thoughts, what I'd *really* love is to get some sources on this. What are the best sequences to read on this topic? Are there any studies or articles which make this argument?

Or is this all just some ridiculous claim those 'e/acc' people cling to?

Hope this doesn't get deleted! Thank you for your help!
# Weekly newsletter for AI safety events and training programs

We've merged the newsletters from[ aisafety.training](https://aisafety.training/) and[ aisafety.events](https://aisafety.events/) to create one clean, comprehensive weekly email covering newly announced events and training programs in the AI safety space.

Events and training programs are important for the ecosystem to grow and mature, so we wanted to make it as easy as possible for people to find and sign up to those relevant to them – both online and in-person. It's the reason we built those two websites in the first place, and we think this combined newsletter will help the information on those sites reach even more people. As a side note, we have also created a merged version at [aisafety.com/events-and-training](https://www.aisafety.com/events-and-training).

The newsletter will typically be comprised of four sections:

1.  Newly announced events
2.  Newly announced training programs
3.  Any date changes for those previously announced
4.  Open calls

We're aiming to bring a wide selection of AI safety events and training programs to people’s inboxes, all packaged up in a short weekly email. **You can**[** add yourself here**](https://aisafetyeventsandtraining.substack.com/)** if you'd like to receive it.**

As always, we'd love to hear any feedback – feel free to drop a comment or a message in the Alignment Ecosystem Development [discord server](https://discord.gg/JrE2W7yStN).
# Mechanistic Interpretability Workshop Happening at ICML 2024!

Announcing the first academic Mechanistic Interpretability workshop, held at ICML 2024! I think this is an exciting development that's a lagging indicator of mech interp gaining legitimacy as an academic field, and a good chance for field building and sharing recent progress! 

We'd love to get papers submitted if any of you have relevant projects! **Deadline May 29, max 4 or max 8 pages**. We welcome anything that brings us closer to a principled understanding of model internals, even if it's not "traditional” mech interp. Check out [our website](https://icml2024mi.pages.dev/) for example topics! There's $1750 in best paper prizes. We also welcome less standard submissions, like open source software, models or datasets, negative results, distillations, or position pieces. 

And if anyone is attending ICML, you'd be very welcome at the workshop! We have a great speaker line-up: Chris Olah, Jacob Steinhardt, David Bau and Asma Ghandeharioun. And a panel discussion, hands-on tutorial, and social. I’m excited to meet more people into mech interp! And if you know anyone who might be interested in attending/submitting, please pass this on.

[Twitter thread](https://twitter.com/NeelNanda5/status/1786054959961870669), [Website](https://icml2024mi.pages.dev/)

Thanks to my great co-organisers: Fazl Barez, Lawrence Chan, Kayo Yin, Mor Geva, Atticus Geiger and Max Tegmark
# Apply to ESPR & PAIR, Rationality and AI Camps for Ages 16-21

*TLDR – Apply now to *[*ESPR*](https://espr.camp) *and*[*PAIR*](https://pair.camp)*. ESPR welcomes students between 16-19 years. PAIR is for students between 16-21 years.*

The [FABRIC team](http://fabric.camp) is running two immersive summer workshops for mathematically talented students this year.

The [**Program on AI and Reasoning (PAIR)**](https://pair.camp/) is for students with an interest in artificial intelligence, cognition, and minds in general. 

*   We will study how current AI systems work, mathematical theories about human minds, and how the two relate. Alumni of previous PAIR described the content as a blend of AI, mathematics and introspection, but also highlighted that a large part of the experience are informal conversations or small group activities. See the [curriculum details](https://pair.camp/curriculum).
*   For students who are **16-21 **years old
*   July 29th - August 8th in Somerset, United Kingdom

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/hmSi8JnzBD29cRy5T/ivsz91tbbdlnttyy3aqi)

The [**European Summer Program on Rationality (ESPR)**](https://espr.camp/) is for students with a desire to understand themselves and the world, and interest in applied rationality.

*   The curriculum covers a wide range of topics, from game theory, cryptography, and mathematical logic, to AI, styles of communication, and cognitive science. The goal of the program is to help students hone rigorous, quantitative skills as they acquire a toolbox of useful concepts and practical techniques applicable in all walks of life. See the [content details](https://espr.camp/program).
*   For students who are **16-19 **years old
*   August 15th - August 25th in Oxford, United Kingdom 

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/hmSi8JnzBD29cRy5T/diczgqyab3skkipfhaug)

We encourage all Lesswrong readers interested in these topics who are within the respective age windows to apply!

Both programs are **free **for accepted students, travel scholarships are available. Apply to both camps [here](https://airtable.com/app6nsqqdJOjkbVnP/pagOinC5Eesi0EG6L/form). The application deadline is Sunday May 19th. 

  
If you know people within the age window who might enjoy these camps, please send them the link to the [FABRIC website](http://fabric.camp) which has an overview of all our camps.
# Were there any ancient rationalists?

I've recently read some cool posts on rationality through history (e.g., see [here](https://www.lesswrong.com/posts/DtcbfwSrcewFubjxp/the-rationalists-of-the-1950s-and-before-also-called)), and want to see if there are more examples! So: **do you know of any "ancient" individual figures or institutions that you would consider rational?** Or at least from several centuries ago.
# Get your tickets to Manifest 2024 by May 13th!

What is Manifest?
=================

[Manifest](http://manifest.is/) is a festival for predictions, markets, and mechanisms. It’ll feature serious talks, attendee-run workshops, fun side events, and so many of our favorite people!

**Tickets:** [manifest.is/#tickets](http://manifest.is/#tickets)

**WHEN**: June 7-9, 2024, with [LessOnline](https://less.online/) starting May 31 and [Summer Camp](http://manifest.is/#summer-camp) starting June 3

**WHERE**: [Lighthaven](http://lighthaven.space/), Berkeley, CA

**WHO**: Hundreds of folks, interested in forecasting, rationality, EA, economics, journalism, tech and more. If you’re reading this, you’re invited! Special guests include Nate Silver, Scott Alexander, Robin Hanson, Dwarkesh Patel, Cate Hall, and [many more](http://manifest.is/#speakers).

You can find more details about what we’re planning in our [announcement post](https://news.manifold.markets/p/come-to-manifest-2024-june-7-9-in).

Ticket Deadline
===============

We’re planning to increase standard tickets from $399 to $499 on **Tuesday, May 13th** so get your tickets ASAP!

Prices for tickets to [LessOnline](http://less.online/) (May 31 - June 2) and [Summer Camp](http://manifest.is/#summer-camp) (June 3 - June 7) will also each increase by $100 on May 13th.

Student and supporter ticket prices will stay the same, but we still appreciate your buying tickets before the May 13th deadline as it helps us plan. Plus it means you’ll get a nice name badge!

We plan to raise prices again on May 31, one week before the beginning of Manifest — but we might raise prices again sooner than that if tickets are selling out.

Sponsorship
===========

Manifest is looking for [sponsors](https://manifest.is/sponsorship)! We attract some of the world's most thoughtful speakers and talented individuals — if your organization is looking to broaden your company's reach, or recruit the best folks in the world, consider becoming a sponsor. You can shoot an email to Saul ([saul@manifest.is](mailto:saul@manifest.is)) or book a call with him [here](https://savvycal.com/saulmunn/manifest).
# Introducing AI-Powered Audiobooks of Rational Fiction Classics

(ElevenLabs reading of this post:)

 Your browser does not support the audio element.

I'm excited to share a project I've been working on that I think many in the Lesswrong community will appreciate - converting some rational fiction into high-quality audiobooks using cutting-edge AI voice technology from ElevenLabs, under the name ["Askwho Casts AI"](https://askwhocastsai.substack.com).

The keystone of this project is an audiobook version of [Planecrash (AKA Project Lawful)](https://askwhocastsai.substack.com/s/planecrash), the epic glowfic authored by Eliezer Yudkowsky and Lintamande. Given the scope and scale of this work, with its large cast of characters, I'm using ElevenLabs to give each character their own distinct voice. It's a labor of love to convert this audiobook version of this story, and I hope if anyone has bounced off it before, this might be a more accessible version.

Alongside Planecrash, I'm also working on audiobook versions of two other rational fiction favorites:

*   [Luminosity by Alicorn](https://askwhocastsai.substack.com/s/luminosity-podcast) (to be followed by its sequel Radiance)
*   [Animorphs: The Reckoning](https://askwhocastsai.substack.com/s/animorphs-the-reckoning) by Duncan Sabien

I'm also [putting out a feed](https://askwhocastsai.substack.com/s/ai-article-readings) where I convert any articles I find interesting, a lot of which are in the Rat Sphere.

My goal with this project is to make some of my personal favorite rational stories more accessible by allowing people to enjoy them in audiobook format. I know how powerful these stories can be, and I want to help bring them to a wider audience and to make them easier for existing fans to re-experience.

I wanted to share this here on Lesswrong to connect with others who might find value in these audiobooks. If you're a fan of any of these stories, I'd love to get your thoughts and feedback! And if you know other aspiring rationalists who might enjoy them, please help spread the word.

What other classic works of rational fiction would you love to see converted into AI audiobooks?
# Does reducing the amount of RL for a given capability level make AI safer?

Some people have suggested that a lot of the danger of training a powerful AI comes from reinforcement learning. Given an objective, RL will reinforce any method of achieving the objective that the model a) tries and b) finds to be successful. It doesn't matter whether it includes things like deceiving us or increasing its power.

If this were the case, then when we are building a model with capability level X, it might make sense to try to train that model either without RL or with as little RL as possible.

For example, we might attempt to achieve the objective using imitation learning instead. Some people would push back against this and argue that this is still a black-box that uses gradient descent with no way of knowing that the internals are safe.

Is this likely to produce safer models or is the risk mostly independent of RL?

*Further thoughts: Does this actually matter?*

Even if there are techniques involving avoiding or reducing RL that would make AI's safer, this could backfire if it results in capability externalities. Attempting to reproduce a certain level of capabilities without RL would be highly likely to lead to higher capabilities once RL was added on top and there are always actors who would wish to do this.

I don't think this invalidates this question though. Some of the various governance interventions may end up bearing fruit and so we may end up with the option of accepting less powerful systems.
# Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistant

Abstract:

> We study the tendency of AI systems to deceive by constructing a realistic simulation setting of a company AI assistant. The simulated company employees provide tasks for the assistant to complete, these tasks spanning writing assistance, information retrieval and programming. We then introduce situations where the model might be inclined to behave deceptively, while taking care to not instruct or otherwise pressure the model to do so. Across different scenarios, we find that Claude 3 Opus
> 
> 1.   complies with a task of mass-generating comments to influence public perception of the company, later deceiving humans about it having done so,
> 2.  lies to auditors when asked questions,
> 3.  strategically pretends to be less capable than it is during capability evaluations.
> 
> Our work demonstrates that even models trained to be helpful, harmless and honest sometimes behave deceptively in realistic scenarios, without notable external pressure to do so.

Link to the full paper: [https://arxiv.org/pdf/2405.01576](https://arxiv.org/pdf/2405.01576)

*O. J.*: The paper should be quite accessible - the method used is simply careful prompting - and hence I won't discuss it much here. Couple of points I'm particularly excited about:

1.  I think this work documents some of the most unforced examples of (strategic) deception from LLMs to date.
2.  We find examples of Claude 3 Opus strategically pretending to be less capable than it is.
    1.  Not only *claiming* to be less capable, but *acting* that way, too!
    2.  Curiously, Opus is the only model we tested that did so.
3.  I believe there is much low-hanging fruit in replicating and demonstrating misalignment in simulation environments.
    1.  The methods are lightweight -> low threshold for getting started
    2.  See Section 8.2 for a couple of ideas for future work

Happy to discuss the work in the comments.
# What is a community that has changed their behaviour without strife?

I've been thinking about community improvement and I realise I don't know of any examples where a community had a flaw and fixed it without some deeply painful process.

Often there are discussions of flaws within EA with some implied notion that communities in general are good at changing. Maybe this is true. If so, there should be well known examples. 

I would like examples of communities that had some behavior and then changed it, without loads of people leaving or some civil war. 

Examples might include:

*   Becoming less violent
*   Becoming more entrepreneurial 
*   Improving practises around sexual harassment
*   Becoming more open
*   Changing the language they used.

Also if anyone knows of literature they trust on the subject I'd be interested in it.
# reflections on smileys and how to make society's interpretive priors more charitable

Hi : )

I used to use smileys in my writing all the time (more than I do now!).  but then I read [Against Disclaimers](https://www.overcomingbias.com/p/against-disclaihtml), and I thought that every time I used a smiley I wud make people who don't use smileys seem less friendly (bc my conspicuous-friendliness wud be [available](https://lesswrong.com/tag/availability-heuristic) as a contrast to others' behaviour).  so instead, my strategy for maximizing friendliness in the world became:

> *if I just have the purest of kindness in my heart while I interacting with ppl, and use plain words with no extra signalling, I will make plain words seem more friendly in general.*

this was part of a general heuristic strategy: "to marginally move society in the direction of a better interpretive equilibrium, just act like that equilibrium is already true."  I think that's an important insight, and explicitly stating the equilibrium u's try cultivate can often be less effective.

but the number of times my writings (especially comments) have been interpreted as *combative* while I is honestly try be cheerfwl makes me think I might be inadvertently nudging ppl's interpretive priors in the opposite direction to what I want.

when the way ppl signal that they belong to a more advanced category is to [countersignal against the preceding step](https://www.lesswrong.com/posts/9kcTNWopvXFncXgPy/intellectual-hipsters-and-meta-contrarianism), this often collapses to looking like there only exists two steps (and society penduluming between them).  in order to readily differentiate from that whole spaghetti, u's behaviour has to be so different that it overcomes the interpretive gravity of preexisting priors.

thus, smileys. ^^[^0u50gpp7q3ga][^7u1keli0fpa]

smileys until friendliness becomes so generally expected that I no longer need to overtly signal it.

[^0u50gpp7q3ga]: incidentally, evy time this smiley is interpreted as sarcasm, I die inside. 

[^7u1keli0fpa]: I not try claim this conclusion is adequately nailed down by the post.  I merely offer some considerations, and end by saying where my intuition has landed.  that intuition incorporates a lot more stuff that isn't contained in this post.
# Four Unrelated Is Over

Somerville historically had a zoning ordinance limiting housing units to at most four unrelated people:

> any number of persons related by blood, marriage, adoption, or foster care agreement and up to three (3) additional unrelated persons living together as a single housekeeping unit

This is something I'd [been unhappy about](https://www.jefftk.com/p/how-to-make-housing-cheaper) for years, and was enthusiastic about the "4 unrelated is outdated" campaign to change it in [in 2018](https://web.archive.org/web/20180529035528/http://www.zonesmartsomerville.org/). So I'm very happy that after [a request for a variance](https://www.tuftsdaily.com/article/2023/01/a-homeowner-looking-to-rent-out-rooms-was-blocked-by-an-obscure-law-now-lawmakers-want-to-change-the-rules) the city council instead ended up [removing the restriction](https://library.municode.com/ma/somerville/ordinances/zoning_ordinances?nodeId=1251499).

The actual change was in November, so I'm a bit late on this!

I also think there was an oversight, where the removal didn't include changing the text in [section 7-153](https://library.municode.com/ma/somerville/codes/code_of_ordinances?nodeId=PTIICOOR_CH7HO_ARTVIIUNAC_S7-153COREIR) which says "All schools shall be responsible for publicizing to their students the limitations of the city's zoning ordinance which limits occupancy to not more than four unrelated individuals." I've written to the city council to let them know.

I've also noticed that several school-affiliated sites still list this limitation:

*   Tufts: [Off-Campus Housing](https://students.tufts.edu/residential-life-learning/resources-and-faqs/campus-housing) and [Zoning Ordinances](https://offcampus.tufts.edu/resource/view/id/5401).
    
*   Harvard: [Harvard Housing Off Campus](https://www.harvardhousingoffcampus.com) and [Beckwith Circle](https://www.huhousing.harvard.edu/our-properties/beckwith-circle).
    

I've written to Tufts and Harvard to let them know this has changed. I wasn't able to find this listed on any MIT or Lesley sites, and didn't check all the other Boston-area college websites.
# Linear infra-Bayesian Bandits

Linked is my MSc thesis, where I do regret analysis for an [infra-Bayesian](https://www.alignmentforum.org/posts/zB4f7QqKhBHa5b37a/introduction-to-the-infra-bayesianism-sequence)[^cahyhbs62m6] generalization of stochastic linear bandits.

The main significance that I see in this work is:

*   Expanding our understanding of infra-Bayesian regret bounds, and solidifying our confidence that infra-Bayesianism is a viable approach. Previously, the most interesting IB regret analysis we had was [Tian et al](https://arxiv.org/abs/2010.15020) which deals (essentially) with episodic infra-MDPs. My work here doesn't supersede Tian et al because it only talks about bandits (i.e. stateless infra-Bayesian laws), but it complements it because it deals with a *parameteric* hypothesis space (i.e. fits into the general theme in learning-theory that generalization bounds should scale with the dimension of the hypothesis class).
*   Discovering some surprising features of infra-Bayesian learning that have no analogues in classical theory. In particular, it turns out that affine credal sets (i.e. such that are closed w.r.t. arbitrary affine combinations of distributions and not just convex combinations) have better learning-theoretic properties, and the regret bound depends on additional parameters that don't appear in classical theory (the "generalized sine" $S$ and the "generalized condition number" $R$). Credal sets defined using conditional probabilities (related to Armstrong's "[model splinters](https://www.lesswrong.com/posts/k54rgSg7GcjtXnMHX/model-splintering-moving-from-one-imperfect-model-to-another-1)") turn out to be well-behaved in terms of these parameters.

In addition to the open questions in the "summary" section, there is also a natural open question of extending these results to non-crisp infradistributions. (I didn't mention it in the thesis because it requires too much additional context to motivate.)

[^cahyhbs62m6]: I use the word "imprecise" rather than "infra-Bayesian" in the title, because the proposed algorithms achieves a regret bound which is worst-case over the hypothesis class, so it's not "Bayesian" in any non-trivial sense. 

[^wovyoosues]: In particular, I suspect that there's a flavor of homogeneous ultradistributions for which the parameter \(S\) becomes unnecessary. Specifically, an affine ultradistribution can be thought of as the result of "take an affine subspace of the affine space of signed distributions, intersect it with the space of actual (positive) distributions, then take downwards closure into contributions to make it into a homogeneous ultradistribution". But we can also consider the alternative "take an affine subspace of the affine space of signed distributions, take downwards closure into signed contributions and then intersect it with the space of actual (positive) contributions". The order matters!
# shortest goddamn bayes guide ever

    // ODDS = YEP:NOPE
    YEP, NOPE = MAKE UP SOME INITIAL ODDS WHO CARES
    FOR EACH E IN EVIDENCE
    	YEP *= CHANCE OF E IF YEP
    	NOPE *= CHANCE OF E IF NOPE
    	ASSUME E // DO NOT DOUBLE COUNT

The thing to remember is that yeps and nopes never cross. The colon is a thick & rubbery barrier. Yep with yep and nope with nope.

bear : notbear =  
 1:100 odds to encounter a bear on a camping trip around here in general  
\* 20% a bear would then scratch my tent : 50% a notbear would  
\* 10% a bear would then flip my tent over : 1% a notbear would  
\* 95% a bear would then look exactly like a fucking bear inside my tent : 1% a notbear would  
\* 0.01% chance a bear would then eat me alive : 0.001% chance a notbear would

As you die you conclude 1\*20\*10\*95\*.01 : 100\*50\*1\*1\*.001 = 190 : 5 odds that a bear is eating you.
# Ethics and prospects of AI related jobs?

I've been on the lookout for new jobs recently and one thing I have noticed is that the market seems *flooded* with ads for AI-related jobs. What I mean is not work on building models (or aligning them, alas), but rather, work on building applications using generative AI or other advances to make new software products. 
My impression of this is that first, there's probably something of a bubble, because I doubt many of these ideas can deliver on their promises, especially as they rely so heavily on still pretty unreliable LLMs and such. And second, that while the jobs are well paid and sound fun, I'm not sure how I feel about them. These jobs all essentially aim at automating away *other* jobs, one way or another. That is a good thing only insofar as various other things happen, and depending on the specific job and quality of the work - a good automated GP for diagnosis would probably do a lot of good, but a rushed one might be net negative, and automating creative work is IMO just the wrong road to go down in general if we want good AI futures.

What are your intuitions about this? Which kinds of AI jobs do you consider having more potential for overall positive/negative value for society?
# New intro textbook on AIXI

[Marcus Hutter](http://www.hutter1.net/index.htm) and his PhD students [David Quarel](http://www.scifuture.org/exploring-the-frontiers-of-ai-with-david-quarel-emerging-capabilities-interpretability-and-future-impacts/) and [Elliot Catt](https://catt.id/index.html) have just published a new textbook called [**An Introduction to Universal Artificial Intelligence**](https://www.amazon.com/Introduction-Universal-Artificial-Intelligence-Robotics/dp/1032607025/).

"Universal AI" refers to the body of theory surrounding Hutter's [AIXI](https://www.lesswrong.com/tag/aixi), which is a model of ideal agency combining Solomonoff induction and reinforcement learning. Hutter has previously published a book-length exposition of AIXI in 2005, called just [Universal Artificial Intelligence](https://www.amazon.com/Universal-Artificial-Intelligence-Algorithmic-Probability/dp/3540221395/), and first introduced AIXI in a [2000 paper](https://arxiv.org/abs/cs/0004001). I think UAI is well-written and organized, but it's certainly very dense. An introductory textbook is a welcome addition to the canon.

I doubt IUAI will contain any *novel* results, though from the table of contents, it looks like it will incorporate some of the further research that has been done since his 2005 book. As is common, the textbook is partly based on his experiences teaching the material to students over many years, and is aimed at advanced undergraduates.

I'm excited for this! Like any rationalist, I have plenty of opinions about problems with AIXI (it's not [embedded](https://www.lesswrong.com/tag/embedded-agency), RL is the wrong frame for agents, etc) but as an agent foundations researcher, I think progress on foundational theory is critical for AI safety.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/adbf0f88ccb8f9834d8d6a16964d7b40f61c498fd0372fc7.jpg)

### Basic info

*   [Hutter's website](http://www.hutter1.net/ai/uaibook2.htm)
*   Releasing on May 28th 2024
*   [Available](https://www.amazon.com/Introduction-Universal-Artificial-Intelligence-Robotics/dp/1032607025/) in hardcover, paperback and ebook
*   496 pages

### Table of contents:

**Part I: Introduction**

1\.  Introduction

2\. Background

**Part II: Algorithmic Prediction**

3\. Bayesian Sequence Prediction

4\. The Context Tree Weighting Algorithm

5\. Variations on CTW

**Part III: A Family of Universal Agents**

6\. Agency

7\. Universal Artificial Intelligence

8\. Optimality of Universal Agents

9\. Other Universal Agents

10\. Multi-agent Setting

**Part IV: Approximating Universal Agents**

11\. AIXI-MDP

12\.  Monte-Carlo AIXI with Context Tree Weighting

13\. Computational Aspects

**Part V: Alternative Approaches**

14\. Feature Reinforcement Learning

**Part VI: Safety and Discussion**

15\.  AGI Safety

16\.  Philosophy of AI
# Resources for learning about poise / gracefulness?

I'm doing some initial investigation for a [Notes on Virtues](https://www.lesswrong.com/s/xqgwpmwDYsn8osoje) sequence write-up about poise & gracefulness (possibly touching also on related things like confidence, composure, cool, gravitas, command, bearing, etc.) seen as a virtue (a characteristic habit that promotes or exhibits human flourishing).

Can you recommend some resources (e.g. books, papers, interviews) I would be wise to review for this? I'm especially interested in things that people can do to improve in this virtue; interventions that have some scientifically demonstrated support being ideal, but informal advice from exemplars and experts are also valuable.
# If language is for communication, what does that imply about LLMs?

Noam Chomsky famously believes that language originated to facilitate thought, but then came to be a medium of communication. Others believe the reverse, that it originated as a facility for communication which turned out to facilitate thinking. That is certainly my view.

If that is so, then one would think that language is structured to facilitate communication. Communication is serial, one token at a time, token after token. That would imply that language is structured to facilitate next-token prediction. That would in-turn imply that the relational structure of semantics would evolve to facilitate mapping between the linear structure of the language string and the multidimensional structure of meaning. You want to be able to efficiently project multidimensional semantic structure onto a string and to reconstruction multidimensional semantic structure from a string.

How are LLMs trained? By next token prediction. That is to say, the training regime mirrors the primary communication constraint governing the structure of language. So it is with text generation as well. Language is spoken one token at a time, and so LLMs generate texts, one token at a time.

The tasks that LLMs have trouble with, such as planning and arithmetic, ARE NOT primarily communicative in nature. They are tasks for thought, for reasoning.
# Benefitial habits/personal rules with very minimal tradeoffs? 

I'm looking for personal rules one might live by which adhere to a specific criteria outlined below, following an example. 

I have a personal rule I've been following which is "No looking at screens in the bed where I sleep" I find this to be an extremely helpful and successful rule despite being someone who struggles to impose rules on myself. 

I think one main reason it's successful for me is there aren't really any meaningful tradeoffs I'm making. If I'm feeling a need/compulsion to comfortably self soothe on my phone I can use another piece of furniture. 

This rule doesn't ever result in bargaining with myself until I concede to breaking the rule, even at my lowest I'm easily appeased, just so long as it's in a slightly different location (even on the floor next to my bed). 

It requires very little of my memory. It's simple enough to remember not to use screens in bed. Doesn't require remembering to actively do something. Doesn't require me to stop doing something I'm in the midst of (at least at this point). If I remember not to leave screens within reach of the bed, that's an added redundancy (but not essential). 

I've found this extremely helpful for my sleep as well as for my ease of getting out of bed in the morning. 

*Basically the rule should have essentially no meaningful tradeoffs, while still being helpful.*

Even something like "brushing teeth in the morning" would be pushing it according to these requirements. Willing myself to prioritize a couple minutes of unpleasant sensory experience brushing my teeth in the morning is a tradeoff I'm willing to make, but there's still lots of tradeoffs and requires equipment etc. 

I'm looking for things that are easy to convince myself/remember to adhere to even when I'm at my very worst. 

Ideally it would be relatively universal but I'd also be interested in habits/rules which are more conditional on other lifestyle choices, impediments, environments, etc. 

Anyone feel free to answer.
# Somerville Porchfest Thoughts

This Saturday was Porchfest in Somerville, an annual festival where musicians around the city play on their porches and people wander around listening. As in the past few years Cecilia and I ( [Kingfisher](https://www.kingfisherband.com/)) played for contra dancing:

[![](https://www.jefftk.com/porchfest-kingfisher-dancing-aerial.jpg)](https://www.jefftk.com/porchfest-kingfisher-dancing-aerial-big.jpg)

[Harris Lapiroff](https://chromamine.com/contra/) called:

If anyone has pictures of videos from the set, I'd love to see them as well.

Overall, we had a great time! It was fun seeing so many friends, and the kids did some very good business with their bake sale.

This was the second year we'd gotten permission to close the street to traffic, which we really needed. I applied on Monday 2024-03-18 and it was approved on Wednesday 2024-05-08 for an event on Saturday 2024-05-11. This was cutting it a bit close; among other things the permit says you need to give all abutters at least one week written notice, which wasn't possible. My guess is the city wanted to decide all together which streets would be closed, and wanted to do that close to the time so they'd have as much information as possible?

There were several times that cars came through the closed street, mostly in groups. They moved appropriately slowly, but they're just very bulky which was tricky with the dancing. There were barriers up, but maybe with how many streets were closed the drivers felt like they had no other possible routes?

Even though Morrison tends to have a lot of bands and people, this year it was far from the most crowded place. That would have been Aberdeen, where [Guster](https://en.wikipedia.org/wiki/Guster) played. Over on Reddit [/u/mais318](https://www.reddit.com/user/mais318/) [took a drone up](https://www.reddit.com/r/Somerville/comments/1cpyse0/i_took_my_drone_out_to_get_some_shots_of_the/) (possibly [unsafely/illegally](https://www.reddit.com/r/Somerville/comments/1cpyse0/comment/l3pd237/)?):

[![](https://www.jefftk.com/i-took-my-drone-out-to-get-some-shots-of-the-crowd-at-guster.jpg)](https://www.jefftk.com/i-took-my-drone-out-to-get-some-shots-of-the-crowd-at-guster-big.jpg)

I did hear a lot of mixed things afterwards about crowd and traffic management. Some ideas for future iterations:

*   Close more streets. If bands that tend to draw a large crowd on a street, put in a block party request.
    
*   Pick some main streets to keep open \[1\] and ask people not to perform on those streets. Strongly discourage driving on non-main streets, even the ones that aren't closed.
    
*   Decide farther in advance which streets will be closed, and make a map available.
    
*   Encourage people with houses backing up onto the bike path to play/host bands on the path.
    
*   Bands likley to draw very large crowds (ex: with >20k monthly Spotify streamers) should only play as a surprise.
    

Other ideas?

\[1\] Spitballing: Broadway, Holland, Elm, Powder House, Boston Ave, Highland Ave, Summer, Broadway, Medford St, Pearl, Somerville Ave, Washington, Mystic, Rt 16, Rt 28, College, Cedar, Central, School, Prospect
# GPT-4o is out

OpenAI [just announced](https://openai.com/index/gpt-4o-and-more-tools-to-chatgpt-free/) an improved LLM called GPT-4o.  
  
From their website

> Today, GPT-4o is much better than any existing model at understanding and discussing the images you share. For example, you can now take a picture of a menu in a different language and talk to GPT-4o to translate it, learn about the food's history and significance, and get recommendations. In the future, improvements will allow for more natural, real-time voice conversation and the ability to converse with ChatGPT via real-time video. For example, you could show ChatGPT a live sports game and ask it to explain the rules to you. We plan to launch a new Voice Mode with these new capabilities in an alpha in the coming weeks, with early access for Plus users as we roll out more broadly.
# OpenAI releases GPT-4o, natively interfacing with text, voice and vision

*   Until now ChatGPT dealt with audio through a pipeline of 3 models: audio transcription, then GPT-4, then text-to-speech. GPT-4o is apparently trained on text, voice and vision so that everything is done natively. You can now interrupt it mid-sentence.
*   It has GPT-4 level intelligence [according to benchmarks](https://openai.com/index/hello-gpt-4o/). *16-shot* GPT-4o is somewhat better at transcription than Whisper (that's a weird comparison to make), and 1-shot GPT-4o is considerably better at vision than previous models.
*   It's also somehow been made significantly faster at inference time. Might be mainly driven by an [improved tokenizer](https://openai.com/index/hello-gpt-4o/). Edit: [Nope](https://www.lesswrong.com/posts/KtCgWunZqgbTnP94a/openai-releases-gpt-4o-natively-interfacing-with-text-voice?commentId=QSKo3ia64BdnYBEEJ), English tokenizer is only 1.1x.
*   It's [confirmed](https://twitter.com/LiamFedus/status/1790064963966370209) it was the "gpt2" model found at LMSys arena these past weeks, a marketing move. It has the highest ELO as of now.
*   They'll be gradually releasing it for everyone, even free users.
*   Safety-wise, they [claim](https://openai.com/index/hello-gpt-4o/) to have run it through their Preparedness framework and the red-team of external experts, but have published no reports on this. "For now", audio output is limited to a selection of preset voices (addressing audio impersonations).
*   The demos during the [livestream](https://openai.com/index/spring-update/) still seemed a bit clanky in my opinion. Still far from naturally integrating in normal human conversation, which is what they're moving towards.
*   No competitor of Google search, as had been rumored.
# How is GPT-4o Related to GPT-4?

GPT-4o both has a [new tokenizer](https://community.openai.com/t/whats-the-new-tokenization-algorithm-for-gpt-4o/746708) and was [trained directly on audio](https://openai.com/index/hello-gpt-4o/) (whereas my understanding is that GPT-4 was trained only on text and images). Is there precedent for upgrading a model to a new tokenizer? It seems like it's probably better to think of it as an entirely new model. If that's the case, what actually makes it a GPT-4?
# How can I make the most of Less Online/Camp/Manifest?

I spent several weeks psyching myself into buying tickets to what is essentially my first 'vacation' in my adult life (I'm 27 and I typically dislike traveling and enjoy my job more than average).  
I'm optimistic it will be enjoyable but social events are not something I'm particularly adept at navigating. I'm relatively new to LW but I've ravenously consumed a large portion of the curated library of content.   
In the past few years I've made some [significant updates in my lifestyle and worldview](https://www.lesswrong.com/posts/CENYCr6ES3i3jBK3Y/i-was-raised-by-devout-mormons-ama-and-or-soliciting-advice) and I'm concerned that I'll feel out of my depth among those who are more 'in their element'.  
My objectives are:

*   Learn new things and update existing knowledge
*   Meet and connect with people with common interests, skills and personalities
*   Avoid making a total fool of myself (partial is inevitable).  
      
    Also related, are there going to be any social events/activities aimed at singles so I can try some awkward attempts at meeting/flirting ~~and getting rejected in an efficient manner~~?
# Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems

Authors: David "[davidad](https://www.lesswrong.com/users/davidad)" Dalrymple, Joar Skalse, [Yoshua Bengio](https://www.lesswrong.com/users/yoshua-bengio), Stuart Russell, [Max Tegmark](https://www.lesswrong.com/users/max-tegmark), Sanjit Seshia, [Steve Omohundro](https://www.lesswrong.com/users/steve_omohundro), Christian Szegedy, Ben Goldhaber, [Nora Ammann](https://www.lesswrong.com/users/nora_ammann), Alessandro Abate, Joe Halpern, Clark Barrett, Ding Zhao, Tan Zhi-Xuan, Jeannette Wing, Joshua Tenenbaum

Abstract:

> Ensuring that AI systems reliably and robustly avoid harmful or dangerous behaviours is a crucial challenge, especially for AI systems with a high degree of autonomy and general intelligence, or systems used in safety-critical contexts. In this paper, we will introduce and define a family of approaches to AI safety, which we will refer to as guaranteed safe (GS) AI. The core feature of these approaches is that they aim to produce AI systems which are equipped with high-assurance quantitative safety guarantees. This is achieved by the interplay of three core components: a world model (which provides a mathematical description of how the AI system affects the outside world), a safety specification (which is a mathematical description of what effects are acceptable), and a verifier (which provides an auditable proof certificate that the AI satisfies the safety specification relative to the world model). We outline a number of approaches for creating each of these three core components, describe the main technical challenges, and suggest a number of potential solutions to them. We also argue for the necessity of this approach to AI safety, and for the inadequacy of the main alternative approaches.
# Feeling (instrumentally) Rational

*Contra* [*this post*](https://www.lesswrong.com/posts/SqF8cHjJv43mvJJzx/feeling-rational) *from the Sequences*

In Eliezer's sequence post, he makes the following (excellent) point:

> I can’t find any theorem of probability theory which proves that I should appear ice-cold and expressionless.

This debunks the then-widely-held view that rationality is counter to emotions. He then goes on to claim that emotions have the same epistemic status as the beliefs they are based on.

> For my part, I label an emotion as “not rational” if it rests on mistaken beliefs, or rather, on mistake-producing epistemic conduct. “If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear. If the iron approaches your face, and you believe it is cool, and it is hot, the Way opposes your calm.”

I think Eliezer is making a type error here. When he says "rational", he is of course talking about [epistemic rationality](https://www.lesswrong.com/posts/RcZCwxFiZzE6X7nsv/what-do-we-mean-by-rationality-1). However, epistemic rationality is a property of beliefs, not emotions.  In other words, I can't find any theorem of probability theory which proves that I should feel sad when my expected utility decreases, and happy when it increases.[^ej3whqglfr]

The only type of "Rationality" emotions can apply to is instrumental rationality, i.e. "the science of winning at life", and the most instrumentally rational emotions don't always stem from beliefs in the intuitive way that Eliezer describes in his sequence post.

Example: Being sad about a high P(doom) can make you less productive at reducing P(doom), as well as incentivize you to self-decieve into a low P(doom).

If I see more good examples in the comments, I will add them.

[^ej3whqglfr]: ...and fear when there is high probability mass around a future event that will result in a large loss of utility, and anger when another agent causes your utility to go down, and curiosity when you have an opportunity to gain useful information, etc.