Harley-ml/lesswrong huggingface dataset:
# Contra Steiner on Too Many Natural Abstractions

Reply To: "[Take 4: One problem with natural abstractions is there's too many of them.](https://www.lesswrong.com/posts/NK4XxyrjFWt83m3dx/take-4-one-problem-with-natural-abstractions-is-there-s-too)"

Epistemic Status
----------------

Unconfident. I don't yet have a firm philosophical grounding on abstractions.

* * *

Unequally Made Abstractions
===========================

Not all natural abstractions are made equal. It seems to me that there are different degrees of "naturalness" of an abstraction. When we say a concept is a natural abstraction, we are saying things like:

1.  The concept is highly privileged by the inductive biases of most learning algorithms that can efficiently learn our universe[^opzq9mioyig]
    *   The more privileged the concept is in aggregate[^h1rax762m2b], the more natural the abstraction is
        *   Learning algorithms given weight inversely proportional to how efficiently they learn our universe
    *   The more learning algorithms privilege the concept, the more natural an abstraction it is
2.  Most efficient representations of our universe contain a simple embedding of the subset
    *   \[The simpler the embeddings of the concept are\]/\[the easier it is to point to embeddings of the concept\] in aggregate[^n5ujgadh9d], the more natural the abstraction is

It follows that the most natural abstractions for a concept (cluster) are the abstractions that we'd expect AI systems to actually use/the abstractions that are most relevant for their decision making.

* * *

What's The Most Natural Abstraction for Human Values?
=====================================================

Consider the subset of "human values" that we'd be "happy" (were we fully informed) for powerful systems to optimise for.

\[Weaker version: "the subset of human values that it is existentially safe for powerful systems to optimise for".\]

Let's call this subset "ideal values".

I'd guess that the "most natural" abstraction of values isn't "ideal values" themselves but something like "[the minimal latents](https://www.lesswrong.com/posts/N2JcFZ3LCCsnK2Fep/the-minimal-latents-approach-to-natural-abstractions) of ideal values".

* * *

Conclusions
===========

I don't think the multitude of abstractions of human values is necessarily as big a stumbling block as Steiner posited.

[^opzq9mioyig]: "Learning our universe" means learning a map/world model of our universe that allows effectively predicting future events.The constraint of efficiency suggests that the learned world model (and inferences via it) should have low data, time, and space complexity (relative to what is attainable for optimal learning algorithms). 

[^h1rax762m2b]: When aggregating across learning algorithms we might want to give algorithms weights that are inversely proportional to how efficient they are in learning our universe. 

[^n5ujgadh9d]: When aggregating across representations of our universe we might want to give representations weights that are inversely proportional to how efficient they represent our universe.
# ChatGPT is our Wright Brothers moment

The Wright Brother’s first flight was a joke by modern flying standards. Even though proving heavier than air flight was viable was monumental, many people at the time could not see how a machine that only can fly a short distance, a few feet off the ground, with the pilot in an uncomfortable position, and very questionable safety with no protection against crashing into the ground, could ever amount to anything.

Despite all these obvious flaws, entrepreneurs and engineers saw the potential. They rapidly iterated on the design, and within 2 decades, airplanes were a decisive advantage in war, they were changing the delivery of goods, they were creeping into commercial travel on the very luxury end of the spectrum. A hundred years later, we have modern marvels like the Airbus A380 and Boeing 747. We have international airports which are practically a world wonder in their operations and interoperability.

ChatGPT is analogous to the Wright Flyer. There were capable LLMs and tons of work in AI prior, but ChatGPT put this all together in a way for the general public to imagine how AI could be a part of their life. For others, they saw a clear villain to fear. But what is the analog in AI for the P51 Mustang? Or the B17? Or global air traffic control systems? Or composite materials and jet engines?

AI is going to have as big an impact or bigger as airplanes did, and a huge amount of infrastructure will need to be built in the process. What should humanity be building?
# Announcing: The Independent AI Safety Registry

Are you an independent AI Safety Researcher who would like to get in touch with other people in the same position? [Fill out](https://forms.gle/Sd2YgsnVqLDdrrSQ9) your location, work details, and collaboration interests, and you'll be added to a public google sheets file where Independent AI Safety Researchers can more easily get in touch with each other.

[Here](https://docs.google.com/spreadsheets/d/1Us_eYfpzM5qg52aZh2Cc8O13Fdfiy5CMKmzZhN5ItOY/edit?usp=sharing) you can browse the entries so far. In a couple of months I'll lock the file and the form, cause there is no updating mechanism for old entries. If the concept of this registry is wildly successful, I'll look into a more persistent format at that point.

Enjoy!

*   [Form](https://forms.gle/eHJ87Hi6ygcn1Mr18) \- So You Can Join
*   [File](https://docs.google.com/spreadsheets/d/1Us_eYfpzM5qg52aZh2Cc8O13Fdfiy5CMKmzZhN5ItOY/edit?usp=sharing) \- So You Can Browse
# What policies have most thoroughly crippled (otherwise-promising) industries or technologies?

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/28849e8d6a186bb440d98e8eb58c2d6801fb2877ff78b363.png)

(cropped from an image by DALL-E 2)

In order to seriously consider promoting policies aimed at slowing down progress toward transformative AI, I want a better sense of the reference class of such policies.

*   What policies do you know of that have "done the most damage" to industry or progress in some restricted domain?
*   (optional) Exactly what did those policies "accomplish" and how? How would you measure their impact?
*   (optional) Was the crippling effect intentional on the part of the policymakers?
# Fractional Resignation

When you're playing games you generally want to spend your time exploring interesting situations, which are typically ones where there are a range of outcomes. If someone is definitely going to win it's not usually much fun to keep going, and in many gaming cultures when you get to this point it's polite to resign. The problem is, what do you do when it's only very likely that someone will win?

In some games, this is still a very interesting area to explore, with lots of interesting choices even for the losing player, but in others (Risk, Go, etc) it would usually be better to stop and play something else. In these cases, bargaining over fractional resignations can work well.

For example, I might think I have an [80% chance of losing](https://www.jefftk.com/p/two-player-catan), and if further play doesn't seem fun I could offer my opponent 80% of a resignation. This is something that we would think of as essentially equivalent to them winning four games and me winning one game. If they think they actually have a 90% chance of winning we could bargain and maybe end up at 85%, but usually that isn't needed.

This does mean that more games end in resignations, but only in cases where that's what everyone wants. It works especially well if you're gaming with someone who doesn't like to resign if there's a chance, but you're playing games where finishing them off can be just really tedious.

In games with more than two players, negotiating fractional resignation should still be possible, but I don't think I've ever done it. It's more common for these games to have dynamics that keep it interesting for everyone, however, so this is often less needed.

(Avoiding the need for this usually looks like keeping the possibility of anyone winning relatively live until the very end, through some combination of randomness, secrecy, and last-shall-be-first dynamics. For example, when I play Modern Art we usually don't know who will win until we count up the score because people's amounts of money are too hard to keep track of, in Ticket to Ride you don't know what sort of progress opponents are making towards their tickets, and in Power Grid the "penalize the apparently leading player" dynamic is a strong equalizer.)
# Rock-Paper-Scissors Can Be Weird

Let's consider a symmetrical, two-player, zero-sum game, where each player must simultaneously choose one of three moves: A, B, or C. After each player choses their move, a winner is randomly selected with odds based on what the two players selected. (Note that because its symmetrical, if both players choose the same strategy, it is guaranteed to be a 50/50 coin flip for who wins).

In the simplest case, one of the moves wins more than 50% of the time against both of the other two moves. In this case, the best strategy is to always select this move.

Now let's consider a rock-paper-scissors type game. Say A beats B with probability 0.6, B beats C with probability 0.6, and C beats A with probability 0.6. In this game, the best strategy is to chose each of the moves A, B, and C with probability 1/3. So far, so good.

Now let's improve move C by making it beat move A with probability 1, leaving the other two fixed (so A beats B 60% of the time, and B beats C 60% of the time). What does the optimal play look like now?

Even if you don't want to fully compute it, notice what your intuition is telling you should happen.

As it turns out, the optimal strategy is to play move A 1/7 of the time, move B 5/7 of the time, and move C 1/7 of the time. Against this strategy, move A wins (0.5)(1/7) + (0.6)(5/7) + (0)(1/7) = 1/2 of the time, move B wins (0.4)(1/7) + (0.5)(5/7) + (0.6)(1/7) = 1/2 of the time, and move C wins (1)(1/7) + (0.4)(5/7) + 0.5(1/7) = 1/2 of the time, so there is no way to beat this strategy: it is optimal.

This was really surprising to me! We improved move C, and somehow people want to play is *less*. Is there a name for this effect?
# Don't feed the void. She is fat enough!

My posts. I write them. I post them. And that's it. In my head, I am not writing them to someone. And I don't expect anything back. There are obvious problems with this model. When I am writing an email to someone, I think very carefully about what to write. When I want to explain something, I think about how to make the other person understand. My mental state in that situation is significantly better at producing good writing.

But blog posts (or really any form of public writing) are about talking to another person, or ideally many! Having the model that you put out a blogpost to get sucked into the void, makes it harder to empathize with the audience. No wonder; the audience does not exist in this model. To be clear, this model is not one that I have consciously adopted, but today I noticed that it is there. And now it needs to go.

Of course, the void model is in my mind for a reason. It is a protection mechanism. If you don't expect anything you won't be disappointed. Maybe this model can be useful. I am not sure. When I started to publically write, I did not know anybody in my target audience. Especially not for AI alignment topics. Better to feed the void than to become dispirited. I would not be writing this if I lost my spirit.

But now I know at least 7 people in my target audience. So why not treat a post like I would treat an email to somebody? Just pick a person and write the post to them! Just as if I were writing a very long message to that person. I have not tried this yet, but I expect this to be a significant improvement. You try too.
# CFP for Rebellion and Disobedience in AI workshop

Hi everyone!

I'm co-organizing a workshop on a really interesting topic that's very relevant for AI safety. We call it "Rebellion and Disobedience in AI".  If you're doing work that could be relevant for us, please submit it! If you have questions or want to discuss the scope of this workshop, feel free to ask on this thread and I'll try to answer.

Full CFP below:

**Call for Participation: Workshop on Rebellion and Disobedience at AAMAS’23**  
 

This workshop will take place on May 29 or 30, 2023, as part of the AAMAS workshop program.

More details can be found on the workshop’s website: 

[https://sites.google.com/view/rad-ai/home](https://sites.google.com/view/rad-ai/home)

RaD-AI agents are artificial agents (virtual or robots) that reason intelligently about why, when, and how to rebel and disobey their given commands. The need for agents to disobey contrasts with most existing research on collaborative robots and agents, where the definition of a “good” agent is one that complies with the commands it is given, and that works in a predictable manner under the consent of the human it serves. However, as exemplified in Issac Asimov’s Second Law of Robotics, this compliance is not always desired, such as when it might interfere with a human’s safety. While there has not been much prior research on RaD-AI, we identify main related topics, each of which is studied by a thriving subcommunity of AI, namely: Intelligent Social Agents, Human-Agent/Robot Interaction, and Societal Impacts. In each of these areas, there are research questions relevant to RaD-AI. 

  
 

**We are specifically interested in submissions in the following topics:**

Intelligent Social Agents (including but not limited to: Goal Reasoning, Plan Recognition, Value Alignment, and Social Dilemmas)

Human-Agent/Robot Interaction (including but not limited to: Human-agent Trust, Interruptions, Deception, Command Rejection, and Explainability) 

Societal Impacts (including but not limited to: Legal and Ethical Reasoning, Liability, AI safety, and AI governance)  
 

**Submission details:**

The submission deadline is **January 20, 2023**.

Notifications will be sent on March 13, 2023.  
The submission website is: [https://easychair.org/cfp/radai23](https://easychair.org/cfp/radai23)

Accepted submission types:

Regular Research Papers (6 to 8 pages)

Short Research Papers (up to 4 pages)

Position Papers (up to 2 pages)

Tool Talks (up to 2 pages)  
 

**Organizing Committee:**

David Aha, Navy Center for Applied Research in AI; Naval Research Laboratory; Washington, DC; USA

Gordon Briggs, Navy Center for Applied Research in AI; Naval Research Laboratory; Washington, DC; USA

Reuth Mirsky, Department of Computer Science; Bar-Ilan University; Israel ([mirskyr@cs.biu.ac.il](mailto:mirskyr@cs.biu.ac.il))

Ram Rachum, Department of Computer Science; Bar-Ilan University; Israel 

Kantwon L. Rogers, Department of Computer Science; Georgia Tech; USA

Peter Stone, The University of Texas at Austin; USA and Sony AI
# Why not spend more time looking at human alignment?

As we approach AGI, we also approach the risk of alignment failure: due to either mismatch between intended goals and specified goals ('outer misalignment'), or mismatch between specified goals and emergent goals ('inner misalignment'), we end up with a catastrophically powerful failure mode that puts us all in a very bad place.

Right now, we don't know what kind of model will lead to AGI. We can guess, but a few years back people didn't have a lot of hope for LLMs, and now look where we're at. A lot of people were deeply surprised. Solutions to alignment failure may be model dependent, and AGI may emerge from something entirely new, or something old which starts to behave in a surprising way when scaled. It's quite hard to know where to start when faced with unknown unknowns. Of course, if you're here, you already know all of this.

These challenges and uncertainties may be relatively novel to the academic research community (in which AI safety has only recently become a respectable area of study), but they are by no means novel to most parents, who also train models (albeit 'wet' ones) via RLHF and inevitably experience alignment issues. Much of the time, the biggies get mostly resolved by adulthood and we end up with humans that balance self-interest with respect for humanity as a whole, but not always. For example, assertiveness, persuasiveness, public speaking ability and self-confidence are common training goals but when alignment failure occurs, you get humans (who often manage to convince the masses to elect them democratically) that are just one button push away from the destruction of humanity, and of the pathological personality type to actually push it. We don't need AGI to see alignment failure resulting in the future of humanity on a knife edge. We're already there.

Since we expect AGI (even superhuman AGI) to exhibit a large number of human-type behaviours because their models will presumably be trained on human output, why do we not spend more time looking at alignment failure in humans, its determinants, its solutions and its warning signs, and see whether those things yield insights into AI alignment failure mechanisms? As any human parent (like me) will tell you, although probably not in quite these words, aligning biological general intelligences is hard, counterintuitive and also immensely important. But, unlike AI alignment, there is a vast body of research and data in the literature behind what works and what doesn't when it comes to aligning BGIs.

So, why is there so little cross-domain reach into human (especially child) psychology and related fields in the AI community?
# Announcing The Filan Cabinet

Happy holidays! Some months ago, I launched a new podcast called [The Filan Cabinet](https://thefilancabinet.com/), but forgot to announce the podcast on this blog. Today, I rectify that mistake.

In some ways, the podcast is similar to [AXRP - the AI X-risk Research Podcast](https://axrp.net/). On that show, I interview AI x-risk researchers about their work, and try to bring their underlying views about AI x-risk research into the open: why do they think what they’re doing matters, and which research avenues do they find more or less promising?

The main difference is that in [The Filan Cabinet](https://thefilancabinet.com/), I talk about whatever I want to talk about, while still maintaining the goal of helping my audience understand my guests’ perspectives. To give you some sense of the show’s range, the first four episodes are about:

*   [Politics, pandemic preparedness, and semiconductor policy](https://thefilancabinet.com/episodes/2022/09/18/1-carrick-flynn.html)
*   [A conservative Presbyterian view of God and Christianity](https://thefilancabinet.com/episodes/2022/09/18/2-wayne-forkner.html)
*   [Cryptocurrency and stablecoins](https://thefilancabinet.com/episodes/2022/09/18/3-ameen-soleimani.html)
*   [The practicalities and politics of paid plasma donation](https://thefilancabinet.com/episodes/2022/11/08/4-peter-jaworski.html)

A secondary goal of the podcast is to give me practice at interviewing well, with the hope that this practice improves [AXRP](https://axrp.net/). With this in mind, I’ve optimized the production process for speed, meaning that I do less research before each interview, and that I do not release transcripts for episodes. With luck, this will enable me to release more frequently without sacrificing too much quality.

If you would like to listen to the show, you can search “The Filan Cabinet” on your podcast app of choice, or just [click here](https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5saWJzeW4uY29tLzQzODA4MS9yc3M) to see it on Google Podcasts. You can also see announcements of new episodes on [this Twitter account](https://twitter.com/thefilancabinet). You should see some new episodes being released in 2023.
# Progress links and tweets, 2022-12-29

**Links**
---------

*   [City size is determined by transportation technology](https://unchartedterritories.tomaspueyo.com/p/transportation-technology-shapes-cities) (Tomas Pueyo)

[![](https://pbs.twimg.com/media/Fkg7L8bVUAAk53v.png)](https://pbs.twimg.com/media/Fkg7L8bVUAAk53v.png)

**Tweets**
----------

*   [Did any sci-fi predict that when AI arrived, it would be unreliable, often illogical, and frequently bullshitting?](https://twitter.com/jasoncrawford/status/1606529499440775168)
*   [Sam Altman: in the next decade the marginal cost of intelligence and of energy will trend to zero](https://twitter.com/jasoncrawford/status/1605436145927458816)
*   [Where to, Mr. Chemist? “To a thousand untouched shores”](https://twitter.com/jasoncrawford/status/1604929605873172480) (Du Pont, 1939)
*   [Longevity escape velocity](https://twitter.com/jasoncrawford/status/1604945935607296000)
*   [Study by Philip Tetlock finds “a pervasive tendency… to see things as getting worse than they really are”](https://twitter.com/jasoncrawford/status/1605982455936143362)
*   [“Scientists may find themselves reporting only successful experiments”](https://twitter.com/jasoncrawford/status/1606066128173748225)

**Quotes**
----------

*   [Sustainability is the disease, people are the cure](https://twitter.com/jasoncrawford/status/1605393589797879808)
*   [For the first 40 years of photography, handheld cameras were impossible](https://twitter.com/jasoncrawford/status/1605391591291719680)
*   [How communication and transportation networks help eliminate famine](https://twitter.com/jasoncrawford/status/1605985137484775426)

**Queries**
-----------

*   [Data on European pipeline gas and storage?](https://twitter.com/Atomicrod/status/1604979510335737857) (@Atomicrod)

**Retweets**
------------

*   [“Abundance is the only cure for scarcity… Everything else merely *allocates* scarcity”](https://twitter.com/samhinkie/status/1605623633455374336) (@samhinkie, quoting @patio11)
*   [Converting coal plants into nuclear with small modular reactors](https://twitter.com/AlecStapp/status/1605039724212588545). But: [Congress ordered the NRC to provide a pathway for new reactors—and they made the process *more* burdensome](https://twitter.com/AlecStapp/status/1604849495446933504) (@AlecStapp)
*   [Photographer uses AI to generate image variations from his own photos](https://twitter.com/jonst0kes/status/1606008683678388229) (@jonst0kes)
*   [If you want home values to go up, you don’t want homes to be affordable](https://twitter.com/JerusalemDemsas/status/1605209647555780610) (@JerusalemDemsas)
*   [The bureaucracy feedback loop, green card edition](https://twitter.com/nabeelqu/status/1604990839528124418) (@nabeelqu)
*   [How Theodor Engelmann demonstrated that chloroplasts are the site of oxygen production in plants](https://twitter.com/strandbergbio/status/1605548862688460800) (@strandbergbio)

[![](https://pbs.twimg.com/media/FkgOLisWIAEsskr.jpg)](https://pbs.twimg.com/media/FkgOLisWIAEsskr.jpg)
# More ways to spot abysses

I liked Ben Kuhn’s [‘Staring into the abyss as a core life skill’](https://www.benkuhn.net/abyss/).

I’d summarize as:

1.  If you are making a major error—professionally, romantically, religiously, etc—it can be hard to look at that fact and correct.
2.  However it’s super important. Evidence: successful people do this well.

This seems pretty plausible to me.

(He has a lot of concrete examples, which are probably pretty helpful for internalizing this.)

His suggestions for how to do better helped me a bit, but not that much, so I made up my own additional prompts for finding abysses I should consider staring into, which worked relatively well for me:

1.  If you were currently making a big mistake, what would it be?
2.  What are some things that would be hard to acknowledge, if they were true?
3.  Looking back on this time from five years hence, what do you think you’ll wish you changed earlier?
4.  If you were forced to quit something, what do you want it to be?
5.  (Variant on 1:) If you were currently making a big mistake that would be gut-wrenching to learn was a mistake, what would it be?
# Worldly Positions archive, briefly with private drafts

I realized it was hard to peruse past [Worldly Positions](https://worldlypositions.tumblr.com) posts without logging in to Tumblr, which seemed pretty bad. So I followed Substack’s instructions to import the archives into [world spirit sock stack](https://worldspiritsockpuppet.substack.com/). And it worked pretty well, except that SUBSTACK ALSO PUBLISHED MY UNPUBLISHED WORLDLY POSITIONS DRAFTS! What on Earth? That’s so bad. Did I misunderstand what happened somehow in my rush to unpublish them? Maybe. But they definitely had ‘unpublish’ buttons, so that’s pretty incriminating.

This seems to have turned out alright for me, since it looks like I just never wrote any drafts that would be too embarrassing to anyone other than myself. And the most embarrassing to myself are probably at the level of bad and abortive poetry. Plus it turned up a few decent drafts to finish, and the adrenaline was a welcome pick-me-up in my current retreat-driven stimulation drought.

Some good bits of the archive (from pre-[WSSP](https://worldspiritsockpuppet.com/worldlypositions.html) times) according to me:

1.  [Mine-craft](https://worldspiritsockpuppet.substack.com/p/mine-craft-20-07-03): the composition of the ego in a procedurally generated sandbox game
2.  [The time I rented a robot baby](https://worldspiritsockpuppet.substack.com/p/claudius-emanuel-g-s-17-10-20)
3.  [Why fiction is more horrifying than war photography](https://worldspiritsockpuppet.substack.com/p/tv-night-realer-photography-19-06-11)
4.  [Home: up and down, colder and warmer](https://worldspiritsockpuppet.substack.com/p/home-up-and-down-colder-and-warmer-20-02-20): miscellanious thoughts on e.g. warmth and coldness, the sincerity of historic advertising, and why negativity is deep
5.  [How I learned to have fun on command](https://worldspiritsockpuppet.substack.com/p/bay-bridge-having-fun-19-06-06) though I rarely remember to do it
6.  [England: Attunement and borders](https://worldspiritsockpuppet.substack.com/p/england-attunement-and-borders-20-01-30), in which I get possible attunement, companionship, and a visa
# Is ChatGPT TAI? 

Do you count ChatGPT and/or other similar systems (e.g. codex) as transformative AI? I would think so since it has a very fast adoption (million users in 5 days) and is apparently sufficiently more useful on some queries than Google that they had to declare "code red" - a possible threat to their core business. If not, why and where do you see the line it would need to cross to be TAI?
# Christmas Microscopy

I got the kids a [cheap little microscope](https://smile.amazon.com/gp/product/B081F64YLB/) for Christmas that attaches to your computer. While we've played with microscopes before, it's much more fun having one where everyone can see at the same time. Here are some things we saw:

A pine needle that we had just broken, with sap leaking out:

[![](https://www.jefftk.com/microscope-pine-needle.jpg)](https://www.jefftk.com/microscope-pine-needle-big.jpg)

The horsehair of a violin bow:

[![](https://www.jefftk.com/microscope-violin-bow-hair.jpg)](https://www.jefftk.com/microscope-violin-bow-hair-big.jpg)

The carbon fiber 'hair' of my [Incredibow](https://www.incredibow.com/):

[![](https://www.jefftk.com/microscope-incredibow-hair.jpg)](https://www.jefftk.com/microscope-incredibow-hair-big.jpg)

A cloth bag, showing how it's woven and looking a lot like a waffle:

[![](https://www.jefftk.com/microscope-cloth-bag.jpg)](https://www.jefftk.com/microscope-cloth-bag-big.jpg)

A section of the bag where Julia had painted a number in white, showing how the paint is on the surface:

[![](https://www.jefftk.com/microscope-cloth-bag-with-white-paint.jpg)](https://www.jefftk.com/microscope-cloth-bag-with-white-paint-big.jpg)

A crayon box, where you can see that the apparently solid colors are actually made with dots of a few different colors:

[![](https://www.jefftk.com/microscope-crayola-box.jpg)](https://www.jefftk.com/microscope-crayola-box-big.jpg)

A fleece glove:

[![](https://www.jefftk.com/microscope-fleece-glove.jpg)](https://www.jefftk.com/microscope-fleece-glove-big.jpg)

A paper bag, showing the fibers:

[![](https://www.jefftk.com/microscope-paper-bag.jpg)](https://www.jefftk.com/microscope-paper-bag-big.jpg)

My wedding ring, looking very scratched despite feeling smooth to the touch. You can see the reflection of the eight LEDs on the microscope:

[![](https://www.jefftk.com/microscope-wedding-ring.jpg)](https://www.jefftk.com/microscope-wedding-ring-big.jpg)

The hook side of velcro, showing that the hooks are loops that have been cut:

[![](https://www.jefftk.com/microscope-velcro.jpg)](https://www.jefftk.com/microscope-velcro-big.jpg)

The hard part of any present like this is remembering to use it when the opportunity comes up again. Next time the kids are curious about something I think would show up well I'll get it out.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0Gtor8dH7XntQKd9vmiTWWFU7i3dmWhxEwWUZvBwzhiEVNS3obNNQgvuYsYr3cvGul), [mastodon](https://mastodon.mit.edu/@jefftk/109604654414052447)*
# What career advice do you give to software engineers?

I am a defendant of the idea that we have already achieved [rudimentary AGIs](https://www.lesswrong.com/posts/mypCA3AzopBhnYB6P/language-models-are-nearly-agis-but-we-don-t-notice-it) with modern LLMs (as much of a hot take this is), and even though the path to superintelligence is going to be difficult and will probably require a few more technical breakthroughs to make more effective use of available data, I don't think this will take us longer than a decade, or 15 years at most.

When I discuss this idea with some of my CS friends and co-workers, about how AI will inevitably replace most software engineering jobs (picture supercharged Github Copilot that can make entire websites and back-end services on command) most of them ask me the obvious follow-up question: *So what can I do about it?* Yeah, I can help with AI safety/alignment progress but *my job is going to disappear no matter what I do*, and probably sooner than many other more 'physically' demanding ones.

I am always left stumped by this $-$ I simply don't know what to tell them, specially to undergraduates that are still full of hope and totally didn't sign up for this dumpster fire. Should I tell them to just continue doing their thing and see what happens? Let fate take its course and hope for the best? This sounds all too happy-go-lucky for my taste. 

I'd like to hear what you guys think about this matter, what do you answer when asked such questions?
# woke offline, anti-woke online

Like most people , I live in many different spheres which only barely interact. My community is very different from my workplace, is very different from the people I hang out with on the internet, is very different from my family.

Whilst my viewpoints don't change depending on who I'm interacting with, the direction my viewpoints point in often flips sign depending on what the other side believes.

For example I was bought up in an ultra orthodox Jewish community, and now live in a more modern, but still mostly orthodox Jewish community.

Growing up I had to fight hard just to convince my friends and family that gay people aren't evil, deserve to be able to publicly admit they're gay, and that conversion therapy is probably a bad idea. I now live in a community where that's not necessary, but I still often have to argue that you should respect people's chosen pronouns as a matter of basic respect, even if you might personally disagree with transgender politics.

Meanwhile when interact with people online I often find myself arguing that it's still ok to be friends with someone even if they personally don't support gay rights, and that it's worth considering the costs and benefits of gender reassignment surgery on an individual basis, rather than jumping into extremely serious surgery blindly.

From the perspective of my friends growing up I'm a bleeding heart liberal. From the perspective of some of my more extreme friends online I'm a cold hearted bigot.

This also applies to many other controversies. In some circles I need to persuade people that "no we shouldn't litter the environment", in others, that replacing plastic items with paper alternatives doesn't actually do much, if anything, to help the environment, and imposes quite a large cost in the process.

I don't really know if I have any major point here, other than suggesting that [all debates are bravery debates](https://slatestarcodex.com/2013/06/09/all-debates-are-bravery-debates/) and that we should target our arguments according to the circles we find ourselves in, rather than spreading the same message to everyone we meet.
# Would it be good or bad for the US military to get involved in AI risk?

Meant as a neutral question. I'm not sure whether this would be good or bad on net:

Suppose key elements of the US military took x-risk from misaligned strong AI very seriously. Specifically, I mean:

*   Key scientists at the Defense Threat Reduction Agency. They have a giant budget (~$3B/year) and are literally responsible for x-risks. Current portfolio is focused on nuclear risks with some biosecurity
*   Influential policy folks at the Office of the Undersecretary of Defense for Policy. Think dignified career civil servants, 2 levels below the most senior political appointees
*   Commander's Initiative Group at USSTRATCOM. Folks who have the commander's ear, tend to be well-respected, and have a huge effect on which ideas are taken seriously

Why this would be good:

*   The military has far more discretionary budget than anyone else in the world. You could multiply the resources dedicated to AI safety research tenfold overnight
*   The military is a huge source of AI risk (in the sense that advancing AI capabilities faster obviously helps the US in competition with China). If key influencers took the risk seriously, they might be more judicious about their capabilities research
*   A key policy goal is preventing the sharing of AI capabilities research. The military is very good at keeping things secret and has policy levers to make private companies do the same
*   The military is a huge source of legitimacy with the general public. And it seems easier than other routes to legitimacy. I think less than 10 key people actually need to be persuaded on the merits, and everyone else will follow suit
*   If the right person agrees, it's literally possible to get one of the best researchers from this community appointed to lead AI safety research for a major government agency, in the same sense that Wernher von Braun led the space program. You just have to be really familiar with the civil service's intricate rules for hiring

Why this would be bad:

*   If someone presents the ideas badly, it's possible to poison the well for later. You could build permanent resistance in the civil service to AI safety ideas. And it's really easy to make that mistake: presentations that work in these agencies are VERY different from what works in the tech community
*   Even if the agency is persuaded, they could make a noisy and expensive but ultimately useless effort
*   A big government project (with lots of middle managers) adds a moral maze element to alignment research
# Pacing: inexplicably good

Pacing—walking repeatedly over the same ground—[often](https://worldspiritsockpuppet.substack.com/p/oxford-circles-and-planes-19-10-29) feels ineffably good while I’m doing it, but then I forget about it for ages, so I thought I’d write about it here.

I don’t mean just going for an inefficient walk—it is somehow different to just step slowly in a circle around the same room for a long time, or up and down a passageway.

I don’t know why it would be good, but some ideas:

1.  _It’s good to be physically engaged while thinking for some reason._ I used to do ‘gymflection’ with a friend, where we would do strength exercises at the gym, and meanwhile be reflecting on our lives and what is going well and what we might do better. This felt good in a way that didn’t seem to come from either activity alone. (This wouldn’t explain why it would differ from walking though.)
2.  _Different working memory setup:_ if you pace around in the same vicinity, your thoughts get kind of attached to the objects you are looking at. So next time you get to the green tiles say, they remind you of what you were thinking of last time you were there. This allows for a kind of repeated cycling back through recent topics, but layering different things into the mix with each loop, which is a nice way of thinking. Perhaps a bit like having additional working memory.

I wonder if going for a walk doesn’t really get 1) in a satisfying way, because my mind easily wanders from the topic at hand and also from my surrounds, so it less feels like I’m really grappling with something and being physical, and more like I’m daydreaming elsewhere. So maybe 2) is needed also, to both stick with a topic and attend to the physical world for a while. I don’t put a high probability on this detailed theory.
# Why don't Rationalists use bidets?

I've been to many Rat&EA group houses, offices, and other venues both official and unofficial around the Bay Area, and I can't recall ever seeing a bidet. And because I think bidets are superior technology, this is pretty confusing to me. Why might this be?

Possible explanations for why this might be
===========================================

Because bidets are expensive? (No)
----------------------------------

Well, bidet attachments can be retrofitted onto nearly any toilet for ~$35 (e.g. [see amazon](https://amzn.to/3VBLFtt)). They aren't difficult to install either, taking only a wrench and less than 20 minutes of time. (I've done it myself once or twice.) And because they can be installed as just an attachment onto an existing toilet, they don't take up any extra space either.

I'd bet that installing a bidet saves time and money on net.

Because bidets aren't actually great? (Doubt it)
------------------------------------------------

For one, bidets are great. But even if you doubt this point personally, consider that bidets only need to be useful to a fraction of people who use a particular bathroom in order for it to be worthwhile. Besides, bidets can be installed in addition to wherever there's toilet paper, so I don't see how a bidet installation can ever be worse than no bidet installation.

I suppose this is a somewhat-likely explanation, but it's still not great.

*I also spent ten minutes googling for examples of how bidets might actually be ineffective or have unwanted side-effects, but I couldn't find anything that seemed credible.*

Bidets aren't intuitive to use if you've never used them before
---------------------------------------------------------------

If you've never used a bidet before (probably because you're American and the rest of America doesn't use them either) and you encounter one in the wild, then it's a little scary and you probably won't be inclined to experiment with it. And because of this and taboo norms around bathroom stuff, then we shouldn't expect bidet adoption to be memetic when we use each other's bathrooms. 

I dunno, **this is probably why, but I still consider it to be an unsatisfying explanation.** It just boils down to "because almost no one else in America uses them".  Aren't Rationalists supposed to be better than this?

My premises here could be wrong
-------------------------------

Maybe Rationalists actually use bidets? Maybe I missed them, or I've just been to all of the wrong venues? (E.g.: A friend who has been in the community for several years longer than me said that he's seen "5+" bidets at Rationalist group houses. But only "5+"??)

It could be something else?
---------------------------

That's every explanation that I've been able to think of. If you have any ideas for why the Rationalist community has failed to adopt this superior technology (heh) please leave a comment!

* * *

**If you don't have a bidet** [**why don't you install one**](https://amzn.to/3VBLFtt)**??** (*Affiliate link: As an Amazon Associate I earn from qualifying purchases heh*.)
# Default Sort for Shortforms is Very Bad; How Do I Change It? 

# Complaint 

As a user of my shortform, I don't find ordering my posts by "Top" useful. I have to always change it to "New" every time; that's a trivial inconvenience.

Often I want to link to shortform content elsewhere or write shortform content that builds on previous work, and my most recent work is generally what I'm looking for. 

My top shortform post is something I made 6 months ago that's of no relevance to my current work (well thinking/learning about alignment). 


My shortform feed is basically like my Twitter feed but admitting longer posts and exclusively for alignment content. Imagine if Twitter sorted tweets on a user's page by most likes instead of most recent tweets. This is what it feels like as a current shortform user. 


# What I Want 

When I go to others' shortform I also want to see their most recent posts and have to manually toggle it.

I want all visitors to my shortform to also see my most recent posts instead of my most upvoted.

In basically all my shortform use cases I want to see the most recent posts. 


# Conclusions 

I think the default sort for shortform should be "New". I think "Top" basically makes no sense and was just a robustly bad decision.

I'm both asking how to change the shortform experience for myself and visitors to my page and suggesting that the default policy should be changed site wide.
# MacArthur BART (Filk)

*To the tune of "MacArthur Park" with apologies to Jimmy Webb.*

BART was never waiting for us, dear  
We ran to catch the train  
But we missed it, our last chance

MacArthur BART is frightening in the dark  
All the lights they flicker on and off  
Someone left their backpack on the train  
I don't think that I should take it  
Might be full or drugs or makeup  
Better leave it for someone else to find  
Oh, no

I can still remember better days  
The trains ran on time  
Never missed a connection  
The tracks, like gleaming rails to better lands  
Our destinations only a few short stops away

MacArthur BART is frightening in the dark  
Hey I think that guy just flipped me off  
He looks like he's lived through better days  
And he might be a little dangerous  
Might try to take our stuff and hang us  
Better leave him and look the other way  
Oh, no

MacArthur's BART is melting in the dark  
I think my acid finally hit  
Can you see that time is stopping too?  
Hey its really not so scary  
It's actually ordinary  
It's just a matter of your point of view  
Oh, no, oh
# How to eat potato chips while typing

Chopsticks.

![eating potato chips with chopsticks](https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fd25d2ed4-4fe6-47f1-aeeb-52b729f06dbc_3024x4032.png)

Sometimes I want to eat snacks while I work, honestly because I have a weirdly moving dread of stretches of work-time containing nothing in any way gratifying to me. Possibly because I realistically anticipate that I will not remain working long under those conditions.

But sadly many good snacks are oily, crumby, or otherwise ill-suited to being touched repeatedly interleaved with touching one’s keyboard. I think I usually deal with this by eating much snack while not typing at all for a bit, but sort of acting as though I think I’m going to, then washing my hands, then going back to the typing for a while, then repeating—largely defeating the purpose. I also get around this issue by getting bubble tea, a snack which can be substantially consumed with your mouth.

I have often vaguely imagined chopsticks helping, but also imagined that they wouldn’t actually. Today I learned that they work well, at least for potato chips and popcorn.

You might think standard Western cutlery would be the first thing to try, but a) it’s not good for hard, crunchy objects, and b) it’s somehow worse to reach to the side with both hands and do a knife-and-fork action than it is to reach with one hand, so if you need multiple implements to manipulate your snack, chopsticks seem superior.

(For more practical eating advice, see [how to cut and chew meat](https://meteuphoric.com/2016/06/26/are-you-missing-universal-human-skills/). I also learned to drink from a can in the last year or so, but am unlikely to write it up unless I learn that this is a problem for anyone else ever.)
# {M|Im|Am}oral Mazes - any large-scale counterexamples?

Raemon's [recent post](https://www.lesswrong.com/posts/pHfPvb4JMhGDr4B7n/recursive-middle-manager-hell) reminded me that I'm unsure how best to think about bureaucracies and apparent organizational friction, and that I keep meaning to explore why something about my intuition rebels against the "Maze" framing.  I've read a bit of economic and public choice theory, and worked in for-profit organizations very large, large and small, and been exposed to (via friends' work) nonprofits medium and small.  I kind of think it's a working equilibrium, that's likely to reoccur or become worse if we disrupt it.  

Most people are ... not particularly great at coordination or even surface-level alignment.  And not all that motivated to spend more energy than needed on abstract concepts like mission or utility.  Hierarchical mechanisms for coordination are lossy and annoying, but the best we know how to do, once we're doing things that take an order of magnitude bigger than Dunbar's number, and require imperfect people (all of us) to spend much of our time and energy doing things we don't particularly enjoy, but that improve someone else's life enough for us to get paid.  "Get paid", of course, is shorthand for "hard-to-fake signal that someone values our behaviors more than they value that money".

Certainly there's a large variance in just how "mazey" an organization is, even within a size and mission-category group (say, comparing local governments of similar-sized populations, or tech companies of a given size range).  But I don't know of any over a few thousand that aren't mazes to some noticeable extent.  Do you?
# Progress links and tweets, 2023-01-04

**Progress Forum**
------------------

*   [Building Fast and Slow Part III: Design of the World Trade Center](https://progressforum.org/posts/JGuTKS2xaPAoobbMt/building-fast-and-slow-part-iii-design-of-the-world-trade) (Brian Potter)
*   [Why pessimism sounds smart](https://progressforum.org/posts/uXcjBQgQRwakxdi28/why-pessimism-sounds-smart) (a oldie by yours truly)

**Announcements**
-----------------

*   [Foresight Institute has a spot for a J1 visa in San Francisco in 2023](https://twitter.com/allisondman/status/1607300173561987072) (@allisondman)

**Tweets**
----------

*   [One container ship carries more than the whole English fleet did 440 years ago](https://twitter.com/jasoncrawford/status/1608860266380529664). Also: [“What, load boxes ashore and then load the boxes on the ship?”](https://twitter.com/johnstretch/status/1608884299415318529)
*   [“A fully general argument against *ever doing anything that changes anything, ever*”](https://twitter.com/jasoncrawford/status/1607802196330180608)
*   [Sometimes giving someone a book changes the course of their life](https://twitter.com/jasoncrawford/status/1608303874527612928)

**Retweets**
------------

*   [“We already have the tools to preserve brains fantastically well”](https://twitter.com/KennethHayworth/status/1607546480302391296)
*   [So many of the world’s great infrastructure projects would be impossible today](https://twitter.com/PalmerLuckey/status/1607596449185005568)
*   [The 1930 campaign to stop people from listening to recorded music](https://twitter.com/drorpoleg/status/1608133865746358282)
*   [Who are some good, interesting, up-and-coming, not-yet-famous essayists/bloggers?](https://twitter.com/visakanv/status/1608499644509851650)
*   [ChatGPT can correct OCR errors in historical texts](https://twitter.com/marquezxavier/status/1607615809274400771)
*   [California court rules that economic growth as such is an environmental harm](https://twitter.com/ProfSchleich/status/1607423543972876291) (!)
*   [A rapid combo test for covid, flu and RSV. Unfortunately illegal in the US](https://twitter.com/Chris_Said/status/1608146903232806913)

[![](https://pbs.twimg.com/media/FlFIKPVWYAEnFLQ.jpg)](https://pbs.twimg.com/media/FlFIKPVWYAEnFLQ.jpg)