Harley-ml/lesswrong huggingface dataset:
# [Link Post] Cyber Digital Authoritarianism (National Intelligence Council Report)

**Disclaimer**: The U.S. government selectively declassified this report for unknown reasons, and anti-Chinese-Communist-Party sentiment is high in Washington D.C. While I doubt this means the contents are factually inaccurate, it is likely framed to highlight dangerous or unethical aspects of CCP activity, and likely suffers from whatever bias one would expect from the National Security community. 

I don't think that anything in this report is very surprising (50% of it is redacted, and it's only 6 pages anyways), but I think it is useful to get a sense of what some members of the national security community are thinking about re: China and digital authoritarianism.

**Highlights**

*   Concern around China and Russia using large scale data analysis to influence or coerce actors in the United States and other countries.
*   "Growing concern in Europe and other democracies about Chinese and Russian cyber actions and personal privacy creates an opportunity to propose alternatives to blunt digital authoritarianism"

**My Takeaway**

*   Maybe folks with an interest in national security topics could do work on understanding the risk of and preventing long-term, stable, AI powered totalitarianism. I'd be excited for more work in that area. 
*   Maybe there could be interest among many stakeholders for AI governance work to help prevent or regulate capabilities that could contribute to digital authoritarianism. I might imagine this could overlap with AI governance efforts to reduce existential or catastrophic risks.
# Bayesian Scenario: Snipers & Soldiers

*You are a soldier, planning to attack an enemy position. You have a foolproof strategy for taking positions held by ordinary enemies, which will get you killed if you use it on positions held by snipers. You can call in an airstrike that will reliably devastate the enemy position, but you'd prefer not to if you don't have to. Using an airstrike will overextend the army's air support; there's a Q% chance this will lead to a fellow soldier's death.*  
 

*X% of enemies are snipers.*

*Snipers hit their target Y% of the time.*

*Ordinary enemies hit their target Z% of the time.*

  
*\[Poke a helmet on a stick out of the trench\]*

*\[Call in an airstrike\] \[Attack!\]*

* * *

I made a miniature interactive [scenario](https://h-b-p.github.io/Snipers-and-Soldiers/) to help people develop and/or evaluate their understanding of Bayes' Theorem. I intend to use it to teach a class a week from now, and I'm hoping to get feedback before then. Help me (and possibly yourself) out?
# Respect Chesterton-Schelling Fences

*This post is inspired by the recent Ziz-revelations posted here and elsewhere.*

[Chesterton Fence](https://wiki.lesswrong.com/wiki/Chesterton%27s_Fence) injunction: Do not remove a protective barrier unless you know why it was put there in the first place.

[Schelling fence](https://www.lesswrong.com/posts/Kbm6QnJv9dgWsPHQP/schelling-fences-on-slippery-slopes) injunction: Do not cross a generally agreed upon guardrail, lest you end up sliding down an increasingly slippery slope, not noticing until it is too late.

I think a term like a Chesterton-Schelling Fence injunction might be useful: Respect an ethical injunction even if you think you know why it was put there in the first place. 

A somewhat simplified example: There is a rather strong Schelling fence against, say, killing someone. Suppose the stated reasoning behind it is "God commanded so". Some day, you deconvert and start questioning the tenets of your faith, throwing one injunction after another, assuming you know why it was there, not realizing that this particular Chesterton fence is fake, the real reason is an unstated Schelling fence that has little to do with religion, but a lot with living in a society.

I said "respect" not "obey", because it is often hard to tell whether there is a hidden Schelling fence behind a Chesterton fence, and how strong the former is. Or vice versa. Or how many of the various hidden fences are there. Is it okay to cheat in an unhappy marriage? Maybe, maybe not, but noticing that this is an unsafe territory, that respecting the societal norms is generally a safe default, and that crossing it is likely yo backfire in both expected and unexpected ways can be quite useful.
# Prediction Thread: Make Predictions About How Different Factors Affect AGI X-Risk.


In this post you can make several predictions for how different factors affect the probability that the creation of AGI leads to an extinction level catastrophe. This might be useful for planning.

Please let me know if you have other ideas for questions that could be valuable to ask.

Elicit Prediction ([forecast.elicit.org/binary/questions/EviuxuEi3](forecast.elicit.org/binary/questions/EviuxuEi3))

Predictions based on who develops AGI:

Elicit Prediction ([forecast.elicit.org/binary/questions/v4fhohmfO](forecast.elicit.org/binary/questions/v4fhohmfO))

Elicit Prediction ([forecast.elicit.org/binary/questions/qDtulAocc](forecast.elicit.org/binary/questions/qDtulAocc))

Elicit Prediction ([forecast.elicit.org/binary/questions/7kqkKdLCX](forecast.elicit.org/binary/questions/7kqkKdLCX))

Elicit Prediction ([forecast.elicit.org/binary/questions/Ji-f9tH1B](forecast.elicit.org/binary/questions/Ji-f9tH1B))

Predictions based on technology used for developing AGI:

Elicit Prediction ([forecast.elicit.org/binary/questions/vEPhQz4Qz](forecast.elicit.org/binary/questions/vEPhQz4Qz))

Elicit Prediction ([forecast.elicit.org/binary/questions/pPq7NHe_C](forecast.elicit.org/binary/questions/pPq7NHe_C))

Prediction based on approach for creating AGI:

Elicit Prediction ([forecast.elicit.org/binary/questions/fcZH5b2m8](forecast.elicit.org/binary/questions/fcZH5b2m8))

Elicit Prediction ([forecast.elicit.org/binary/questions/ZvtW_KBSR](forecast.elicit.org/binary/questions/ZvtW_KBSR))

Predictions on how money affects probability of AGI X-risk:

Elicit Prediction ([forecast.elicit.org/binary/questions/yAooUTWqA](forecast.elicit.org/binary/questions/yAooUTWqA))

Elicit Prediction ([forecast.elicit.org/binary/questions/Aa7i8eqn1](forecast.elicit.org/binary/questions/Aa7i8eqn1))
# Beginning to feel like a conspiracy theorist

I see myself, and is often described by others, as a logical person and high in the openness trait. I am in university and have the last 2 years done some various amateur litterature review reasearch on different topics, especially on health and nutrition. 

On many topics i find my research agreeing with conventional wisdom, such as for: 

*   Sleeping hygiene
*   Alcohol consumption
*   Drugs like heroin and methamphetamine
*   Benefits of training
*   Dangers of obesity

But on a worrying fast increasing list i find my reasearch contradict common wisdom in huge ways:

*   [Carbhydrates are probably what should be limited, and not fats](https://en.wikipedia.org/wiki/Good_Calories,_Bad_Calories)
*   [Following the WHO recommendations for salt intake is maybe way to little and potentially deadly](https://www.nejm.org/doi/full/10.1056/nejmoa1311889)
*   [Facemasks don't limit the spread of COVID-19 and other similar influensas on a population level](https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD006207.pub6/full)
*   Psychedelics are not addictive and can probably be helpful in many cases. (I think acceptance for this is rapidly increasing)

I don't have any problem trusting data and research more than authorities and common wisdom. But when friends and family significantly express their concern for my well being it makes feel like i am one of those conspiracy theorists that believe every stupid thing they find online. It may be that I am not having a strong enough prior, before updating it with new evidence.
# A case for capabilities work on AI as net positive 

Edit: Changed the title.

Or, why I no longer agree with the standard LW position on AI anymore.

In a sense, this is sort of a weird post compared to what LW usually posts on AI.

A lot of this is going to depend on some posts that changed my worldview on AI risk, and they will be linked below:

Deceptive alignment skepticism sequence, especially the 2nd post in the sequence is here: 

https://www.lesswrong.com/posts/RTkatYxJWvXR4Qbyd/deceptive-alignment-is-less-than-1-likely-by-default

Evidence of the natural abstractions hypothesis in action:

https://www.lesswrong.com/posts/BdfQMrtuL8wNfpfnF/natural-categories-update

https://www.lesswrong.com/posts/obht9QqMDMNLwhPQS/asot-natural-abstractions-and-alphazero#comments

Summary: The big updates I made was that deceptive alignment was way more unlikely than I thought, and given that deceptive alignment was a big part of my model of how AI risk would happen (about 30-60% of my probability mass was on that failure mode), that takes a big bite out of the probability mass of extinction enough to make increasing AI capabilities having positive expected value. Combine this with the evidence that at least some form of the natural abstractions hypothesis is being borne out by empirical evidence, and I now think the probabilities of AI risk have steeply declined to only 0.1-10%, and all of that probability mass is plausibly reducible to ridiculously low numbers by going to the stars and speeding up technological progress.

In other words, I now believe a significant probability, on the order of 50-70%, that alignment is solved by default.

EDIT: While I explained why I increased my confidence in alignment by default in response to Shiminux, I now believe that for now I was overconfident on the precise probabilities on alignment by default.

# What implications does this have, if this rosy picture is correct?

The biggest implication is that technological progress looks vastly positive, compared to what most LWers and the general public think. 

This also implies a purpose shift for Lesswrong. For arguably 20 years, the site was focused on AI risk, though it arguably exploded with LLMs and actual AI capabilities being released.

What it will shift to is important, but assuming that this rosy model of alignment is correct, then I'd argue a significant part of the field of AI Alignment should and can change purpose to something else.

As for Lesswrong, I'd say we should probably focus more on progress studies like Jason Crawford and inadequate equilibria and how to change them.

I welcome criticism and discussion of this post, due to it's huge implications for LW.
# The Birth and Death of Sydney — The Bayesian Conspiracy Podcast

We discuss Microsoft’s new Bing search chatbot – Sydney. It received a wave of media coverage due to erratic behavior, before being neutered by Microsoft. A dive into its behavior, human responses, and implications.

**Links**

[The Birth and Death of Sydney](https://deathisbad.substack.com/p/the-birth-and-death-of-sydney)  
[“Sydney Is Now Immortal”](https://deathisbad.substack.com/p/sydney-is-now-immortal-602)  
[Zvi’s Round-up](https://thezvi.substack.com/p/ai-1-sydney-and-bing)  
[Gwern’s comment on Bing/Sydney](https://www.lesswrong.com/posts/jtoPawEhLNXNxvgTT/bing-chat-is-blatantly-aggressively-misaligned?commentId=AAC8jKeDp6xqsZK2K)  
[Astral Codex Ten post on Janus Simulators](https://astralcodexten.substack.com/p/janus-simulators)  
[Bankless crypto podcast – We’re All Gonna Die with Eliezer Yudkowsky](https://www.youtube.com/watch?v=gA1sNLL6yg4&ab_channel=BanklessShows)  
[Eneasz’s Change.org Petition](https://www.change.org/p/unplug-the-evil-ai-right-now)  
[Sydney-based erotica](https://amzn.to/3KolxA9)

**Other Links**  
[The Parable of the Dagger](https://www.thebayesianconspiracy.com/wp-content/uploads/2023/The_Parable_of_the_Dagger.mp3) and [The Parable of the Dagger](https://www.thebayesianconspiracy.com/wp-content/uploads/2023/synthesized_audio.mp3). One of these is Eneasz, the other one is a voice-matching AI.  
[The Uncanny Vulva](https://dianaverse.com/2020/10/30/uncanny-vulvas/)  
[Follow Moridinamael on Twitter](https://twitter.com/moridinamael)

email the show: bayesianconspiracypodcast@gmail.com

Which AI do you prefer:

[![](https://www.thebayesianconspiracy.com/wp-content/uploads/2023/02/Sydney-zoom-300x297.jpg)](https://www.thebayesianconspiracy.com/wp-content/uploads/2023/02/Sydney-zoom.jpg)

or

[![](https://www.thebayesianconspiracy.com/wp-content/uploads/2023/02/Shoggoth.png)](https://www.thebayesianconspiracy.com/wp-content/uploads/2023/02/Shoggoth.png)(Spoiler: they’re the same picture)

0:00:53 – Steven’s Media Corner  
0:01:48 – Chat with Eneasz IRL  
0:02:31 – Sydney/Bing AI  
01:31:09 – [Guild of the Rose](https://guildoftherose.org/) Update  
01:33:58 – Thank the Patron

* * *

We now partner with [The Guild of the Rose](https://guildoftherose.org/), check them out.  
Hey look, we have a [discord](https://discord.gg/Vwskrh7cZj)! What could possibly go wrong?  
[Our Patreon page](https://www.patreon.com/BayesianConspiracy)–your support is most rational, and totally effective. ([also merch](https://www.redbubble.com/people/BayesConsp/explore?asc=u&page=1&sortOrder=recent))

* * *

[Rationality: From AI to Zombies, The Podcast](https://player.fm/series/rationality-from-ai-to-zombies)

**LessWrong Sequence Posts Discussed in this Episode:**

still not

**Next** [**Sequence Posts**](https://wiki.lesswrong.com/wiki/Less_Wrong/All_articles)**:**

[The Allais Paradox](http://lesswrong.com/lw/my/the_allais_paradox/)

[Zut Allais!](http://lesswrong.com/lw/mz/zut_allais/)
# $20 Million in NSF Grants for Safety Research

After a year of negotiation, the NSF has [announced](https://beta.nsf.gov/funding/opportunities/safe-learning-enabled-systems) a **$20 million** request for proposals for empirical AI safety research.

[Here is the detailed program description](https://www.nsf.gov/pubs/2023/nsf23562/nsf23562.pdf).

The request for proposals is broad, as is common for NSF RfPs. Many safety avenues, such as transparency and anomaly detection, are in scope:

*   "reverse-engineering, inspecting, and interpreting the internal logic of learned models to identify unexpected behavior that could not be found by black-box testing alone"
*   "Safety also requires... methods for monitoring for unexpected environmental hazards or anomalous system behaviors, including during deployment."

Note that research that has [high capabilities externalities](https://www.alignmentforum.org/posts/dfRtxWcFDupfWpLQo/perform-tractable-research-while-avoiding-capabilities#Safety_capabilities_tradeoffs) is explicitly out of scope:

"Proposals that increase safety primarily as a downstream effect of improving standard system performance metrics unrelated to safety (e.g., accuracy on standard tasks) are not in scope."

Thanks to OpenPhil for funding a portion the RfP---their support was essential to creating this opportunity!
# Ethical and incentive-compatible way to share finances with partner when you both work?

My partner and I have been living together for a year. We currently share our finances to some extent (we have our own bank accounts, but have a shared credit card). I've been thinking about how our current system can be made more optimal. We mostly buy food, travel expenses, and other shared goods on the credit card.

Obviously, any system of sharing finances should leave both partners feeling like the arrangement is fair. That's what I mean by ethical. I think that a relationship should be a team effort, and if you can give your partner 1 utilon at the cost of 0.5 utilons to yourself, you should do it every time. If they do the same thing, I think it's very likely that you're both better off in expectation.

Also, I think it's better if a method of splitting finances as incentive-compatible as possible. By that, I mean, both partners should be incentivized to spend money in a way that optimizes total utility between both partners. Money is the one of the most common causes of arguments in relationships, and it seems to me that more incentive-compatible ways of sharing income might be a way to prevent those arguments.

(I think it was William Vickrey who said something like (paraphrasing) "We should get the part of our economy to that can be made to run off self-interest running as smoothly as possible, so more of our attention can be spent on areas that truly cannot be solved except through altruism." By the way, if anyone can track down the source of this quote, I'd be very grateful.)

Brainstorming (trying to say as much as I can about the problem before proposing a solution):

1.  Diminishing marginal utility means that inequality of consumption is a sign that your relationship could be better. In my case, I make about 4x more than my partner. It would be very bad for aggregate utility if I lived large while my partner had to pinch pennies.
2.  A strategy like "one partner pays for rent, the other pays for food" gives the partner who pays for rent no incentive not to spend on food frivolously. Even if they're not spending frivolously, it seems like the possibility that they are might create relationship friction. ("Ugh, there they go again, ordering uber eats again on my dime.")
3.  Money may be fungible but human brains like hypothecation. Saying "A covers 100% of rent" feels different than "A covers 50% of rent + their partner's gym membership and funko pop addiction" even if the total amount spent is the same.
4.  Related to the previous, it might be nice to have shared financial goals, like a joint bank account that both partners contribute to. Even if it's no different from one partner contributing entirely to the shared account and the other contributing to something else, it feels like less of a team effort.
5.  Having to ask your partner for permission to buy something can feel like an impingement on your autonomy.

Anyone have any other ideas?
# Contract Fraud

The older kids have been playing with the concept of contracts, which has often involved attempts to trick the other into signing something. Like the time when Anna (6y) couldn't read yet and Lily (8y) prepared an unfavorable contract and "read" it to her by making up something with much more acceptable terms. We talked about how a contract requires both people to understand the agreement but it seems not to have stuck.

Yesterday the kids started exploring various forms of contract fraud. Anna wrote a very one-sided attempt at a contract, and didn't consider Lily's unwillingness to sign it to be an issue:

[![I Lily Wise will let Anna have whatever she wants from me](https://www.jefftk.com/anna-contract-fraud.jpg)](https://www.jefftk.com/anna-contract-fraud-big.jpg)

That's Anna forging Lily's signature. I explained that if you write someone else's signature it doesn't count, but it turns out this wasn't the right level of explanation. Lily got Anna to sign a blank piece of paper (an "I want your autograph" should have been very suspicious given the earlier contract shenanigans) and then pasted it onto a contract:

[![I Anna will do whatever Lily and Nora want](https://www.jefftk.com/lily-contract-fraud.jpg)](https://www.jefftk.com/lily-contract-fraud-big.jpg)

We talked about how that's also fraud, but they weren't very interested in my explanation.

I also wanted to get into how in addition to lacking mutual assent these contracts lacked mutual compensation and were probably substantively unconscionable, but they went off to play something else so I explained it to Nora (1y).

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0k7h8iB1rJBe7GVhcpGP81wo7XfrAQWeqgZphmvWEXXNyaawYhSnE571MUvGR7gfql), [mastodon](https://mastodon.mit.edu/@jefftk/109945802591671711kids)*
# Sunlight is yellow parallel rays plus blue isotropic light

When you look up the color temperature of daylight, most sources will say 6500K, but if you buy an LED with that color temperature, it will not look like the sun in the sky. It will seem bluer (or, less yellow-y). Yet, 6500K is arguably the correct number. What is going on?

The answer is Rayleigh scattering. What we perceive as sun rays on the Earth’s surface has traveled through a lot of atmosphere, whereby a lot of the blue light has been scattered away, such that the sun rays look a bit yellowish. But the blue light isn’t all lost! A lot of it does arrive on the surface eventually, but it does so as diffuse light – light that seems to come from the whole sky. Which is of course why the sky looks blue (if it isn’t blocked by clouds).

The result is that the parallel rays that come from the bright spot in the sky (aka, the sun) are (slightly) yellow, but as they mix with the diffuse blue light from the sky, they form white light with a color temperature of about 6500K. More specifically, the [daylight standard D65](https://en.wikipedia.org/wiki/Illuminant_D65) is defined as

> the average midday light in Western Europe, comprising both direct sunlight and the light diffused by a clear sky.

And that’s where we get the 6500K color temperature.

(The sun as seen from space actually has a color temperature of 5800K to 5900K. I haven’t looked into why these numbers are different but I assume it’s because of absorption in the atmosphere.)

But most of the time, not everything is illuminated evenly. The parallel rays from the sun produce shadows and they are *blue*:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/76d5c9dfd9e24da1306e0e96a7a38572f8effbd0f0c21baf.jpg)

([Source](https://www.lonelyplanet.com/articles/top-things-to-do-in-los-angeles). This picture was not taken at midday, so the sun is extra yellow, but this allows us to clearly see the blue shadows.)

So, in order to imitate daylight more faithfully indoors, I think you should separate the components of the light as our atmosphere does. So, use spotlights of yellowish light together with diffuse color-of-the-sky blue light that seems to come from everywhere, to produce blueish shadows.

[Here is a video](https://www.youtube.com/watch?v=6bqBsHSwPgw) of someone doing this the hard way with actual Rayleigh scattering – to very impressive results – but I think you could just buy yellowish LED spotlights (with color temperature of 3500–4500K or so? I tried to find the color temperature of just the sun rays without the skylight, but couldn’t find anything) and blue LED strips mounted on all walls close to the ceiling. And of course, the lights should be very very bright in order to imitate the sun.

I suspect that the blueish shadows will make any white light feel much more pleasant.

I intend to do this myself for my office, but it will take some re-configuration of the lights, so it might take a while and I wanted to get this post out early. If someone manages to try this out before me, I would be very interested to hear about their experiences.
# Taboo "compute overhang"

There's no consensus definition of "compute overhang" or "hardware overhang." See e.g. [1](https://aiimpacts.org/hardware-overhang/) [2](https://www.lesswrong.com/tag/computing-overhang) [3](https://www.lesswrong.com/posts/N6vZEnCn6A95Xn39p/are-we-in-an-ai-overhang) [4](https://arbital.com/p/advanced_agent/#:~:text=Hardware%20overhang%20describes%20a%20situation%20where%20the%20initial%20stages%20of%20a%20less%20developed%20AI%20are%20boosted%20using%20vast%20amounts%20of%20computing%20hardware%20that%20may%20then%20be%20used%20more%20efficiently%20later.) [5](https://www.lesswrong.com/posts/W6wBmQheDiFmfJqZy/brain-inspired-agi-and-the-lifetime-anchor#2__Thesis_and_outline), and I've recently seen researchers use several different definitions. And I asked some friends what "hardware overhang" means and they had different responses (a plurality said it means *sufficient hardware for human-level AI already exists*, which is not a useful concept). If you say "compute overhang" without clarification, many people will misunderstand you.

Instead of tabooing it, we could declare a canonical definition. I think the best candidate is something like: there is a *compute overhang* to the extent that *the largest training runs could quickly be scaled up*. But I think it's probably better for people to avoid the term or define it whenever they use it.

P.S. You should also define "takeoff speed," "transformative AI," and "warning shot" whenever you're using them in a precise way; their usage varies a lot too. (Using those terms to vaguely gesture at "the speed of progress around human-level AI," "powerful AI," and "legible scary AI event," respectively, seems ok).
# Extreme GDP growth is a bad operating definition of "slow takeoff"


I gather that the historical definition of slow takeoff is something akin to "grand GDP growth for several months in a row, due to AI advancements". 

While recent advancements by OpenAI et. al. are impressive, sometimes people will claim they would never have awarded bayes points to slow takeoff for ChatGPT, because one of their primary contentions is that governments will prevent any AI that isn't *politically* revolutionary from making innovations in things like housing or medicine.

If this is true, perhaps there should be a new term, since it still seems useful to talk about a tech tree where we have very strong tool AIs, even if they're not making everybody rich. It seems unlikely to me that the same forces that might prevent AIs from being used to innovate housing will also prevent those AIs from being deployed to AI safety efforts.
# Live Kingfisher Album?

I'm pretty happy with how [Kingfisher](https://www.kingfisherband.com/) sounds at this point: with fifteen gigs since dances restarted and eight in the last six months I feel like our playing has gelled a lot. I've been [recording our dances](https://www.jefftk.com/p/record-your-playing) and listening back there's a lot of good stuff. Which has me wondering about making a live album, But how do people do that these days now that interest in buying CDs at dances is much lower?

I've been seeing bands doing Kickstarters for their albums, in a way that seems a lot like a pre-sale? (Ex: [Great Bear](https://www.kickstarter.com/projects/andrewvannorstrand/great-bears-magic-fantasy-dream-dance), [Buddy System](https://www.kickstarter.com/projects/julievallimont/window-seat), [Pete's Posse](https://www.kickstarter.com/projects/petesposse/petes-posse-project-support-join-our-extended-posse), [Nova](https://www.kickstarter.com/projects/nova-trio/novas-second-album), etc.) Aside from Kickstarter taking a [cut (~9%)](https://www.kickstarter.com/help/fees) are there downsides to the platform? Anyone have good or bad experiences to share?

(In 2012 when the [Free Raisins](https://www.freeraisins.com/) made [a live cd](https://www.freeraisins.com/cd) ([details](https://www.jefftk.com/p/free-raisins-cd-live-summer-2012), [economics](https://www.jefftk.com/p/the-economics-of-producing-your-own-cd)) we picked our favorite sets as a band and then [Audrey](http://www.audreyknuth.com/) (who is also a recording engineer) edited them down to listening length. We printed physical copies to sell at dances, and also put it on bandcamp for people who wanted downloads. By 2014 when we made our [second CD](https://www.freeraisins.com/out-of-the-box) ([details](https://www.jefftk.com/p/preparing-for-our-cd), [economics](https://www.jefftk.com/p/the-economics-of-a-studio-cd)) people were starting to buy fewer CDs, but since a lot of people drive to dances and have CD players in their cars we broke even with 483 sales in the first sixteen months. We also used CD Baby to make the album available on streaming platforms (ex: [Spotify](https://open.spotify.com/artist/1yZpyuhXA1caAApEVeTmlz), [YouTube Music](https://music.youtube.com/channel/UCwJ-1NazTaj8P9xkc9eqZdQ)) which has been bringing in about $100/y. Which isn't nothing but also wouldn't be enough to fund even the mixing for an album.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0vBCjuVqtFooVbq6nRnZPF1VnrZJ8S9uWatiHFQkfJZTwfv5xu4SNf79fy9eUB2nPl), [mastodon](https://mastodon.mit.edu/@jefftk/109951579996525142)*
# Payor's Lemma in Natural Language

*Preceded by:* [*Modal Fixpoint Cooperation without Löb's Theorem*](https://www.lesswrong.com/posts/2WpPRrqrFQa6n2x3W/modal-fixpoint-cooperation-without-loeb-s-theorem)

It turns out Payor's Lemma and its proof can be explained in natural language even more easily than Löb's Theorem.  Here's how.

Imagine a group of people, and let $x$ denote the statement "everyone in the group cooperates".   Payor's Lemma says the following:

**Lemma:** If $\vdash \Box (\Box x \to x) \to x$, then $\vdash x$

First, let's unpack the meaning of the assumption in words:

*   "$\Box x$" may be interpreted as saying "the group verifies (on the basis of logic) that it will cooperate" or "cooperation is believed".  
*   "$\Box x \to x$" is a statement of trustworthiness: if the group verifies that it will cooperate, then it actually will cooperate.  Because a formal verifier can have bugs in it — or, because a group of people can fail to understand itself — this is a non-trivial claim about the group.
*   "$\Box (\Box x \to x)$" says "the group verifies that it's trustworthy" (in the specific sense of trustworthiness above).
*   "$\Box (\Box x \to x) \to x$" says "the group will cooperate on the basis of verified trustworthiness", i.e., "if the group verifies that it's trustworthy, then it will cooperate".
*   "$\vdash \Box (\Box x \to x) \to x$" says "it's verified that the group will cooperate on the basis of verified trustworthiness"

Now let's work through the proof in words, too!  I'll omit saying "it's verified that..." each time, which is what $\vdash$ means.

1.  $\vdash x \to (\Box x \to x)$, by tautology ($A \to (B \to A)$).  This says:  
    "If the group cooperates, then it's trustworthy" (in the specific sense of trustworthiness about cooperation defined above).  
     
2.  $\vdash \Box x \to \Box(\Box x \to x)$, from 1 by $\Box$ necessitation and distributivity.  This says:  
    "If the group verifiably cooperates, it's verifiably trustworthy."  
     
3.  $\vdash \Box(\Box x \to x) \to x$, by assumption.  This says:  
    "Assume the group will cooperate on the basis of verified trustworthiness."  
     
4.  $\vdash \Box x \to x$, from 2 and 3 by modus ponens.  This says:  
    "The group is trustworthy."  
     
5.  $\vdash \Box(\Box x \to x)$, from 4 by $\Box$ necessitation.  This says:  
    "The group is verifiably trustworthy."  
     
6.  $\vdash x$, from 5 and 3 by modus ponens.  This says:  
    "The group cooperates."

Continuing to use "trustworthiness" in the sense above, the whole proof may be summarized as follows: 

"If a group verifiably cooperates, it's verifiably trustworthy (to itself).  Assume the group cooperates on the basis of verified trustworthiness.  Then, it also cooperates on the basis of verified cooperation (a stronger condition), which is what trustworthiness means.  Therefore, the group is trustworthy, hence verifiably trustworthy (assuming we concluded all this using logic), hence the group cooperates (by the assumption)."
# [Fiction] The boy in the glass dome

A boy is sitting inside a glass dome. It would just barely have enough space for him to stand.

It’s night, and he’s watching the stars. He has been doing that for a long time.

There’s a small tube attached to the bottom of the dome. A miniature cargo train runs through that tube, bringing with it small bubbles of nutrients and oxygen. When the train reaches the dome, the bubbles float up in the air and pop, keeping the boy alive.

The tube is much too small for the boy to even get his hand inside, let alone leave that way.

There’s a civilization that’s keeping him in the dome as insurance. If something bad were to happen to the civilization, he at least would stay alive.

The boy looks up at the stars and imagines what it would be like:

Something would fall from the sky. A swarm of meteorites, maybe, or nuclear warheads from another planet. They would strike everything on this planet, utterly devastating all life and leave a dead world behind.

But his dome would keep him safe.

And then he would stand on his feet and grow, breaking the dome from the inside. He would grow from a boy to a man, into a giant with an angel’s wings and a flaming sword. He would walk among the devastated landscape, his enormous feet crushing rubble beneath them.

He would wonder about flying to space, to get his revenge on the people who attacked his world. But by this point, they would be gone. He wouldn’t be able to find them anymore, not in the entire vastness of space.

But then… he would notice something moving under the rubble.

He would lean down to dig it up, and see green life growing from underneath the rubble. Survivors who had hidden for long enough for the invaders to think they were all dead.

The boy – now a man – would help the survivors come out. And vast green vines would make their way from under the rubble, wrap themselves around him and anchor him on the ground, use him as a stalk for growing all the way into space.

They would grow all the way to the planet of the attackers. The vine would wrap itself around that enemy planet, borrowing the boy-man’s strength to squeeze it apart like one might crush an orange in one’s hand.

At first the orange would spill its wet sticky mass on the hand, only to then dry up and turn into dust, blown away by a stellar wind.

The boy in the dome blinks. He realizes that waiting for all of this to happen is not the only option. There is also another way.

He already has the strength to grow into a giant angel. All he needs to do is to stand up and let it happen.

And if he were to do that, and his planet were to be attacked – he could use his sword to deflect the attack. Hit each of the falling meteorites or nuclear warheads like one hits a ball with a bat, send them back into the planet of the attackers. Let it be blown up right away, rather than letting the boy’s planets be destroyed first.

The boy nods. That seems better.

He stands up, and lets himself grow into an angel right away, shattering the dome above him.
# Robin Hanson’s latest AI risk position statement

“While I’ve written on this many times before, it seems time to restate my position.”

“While I agree that this is a logically possible scenario, not excluded by what we know, I am disappointed to see so many giving it such a high credence, given how crazy far it seems from our prior experience. Yes, there is a sense in which the human, farming, and industry revolutions were each likely the result of a single underlying innovation. But those were the three biggest innovations in all of human history. And large parts of the relevant prior world exploded together in those cases, not one tiny part suddenly exterminating all the rest.

In addition, the roughly decade duration predicted from prior trends for the length of the next transition period seems plenty of time for today’s standard big computer system testing practices to notice alignment issues. And note that the impressive recent AI chatbots are especially unlike the systems of concern here: self-improving very-broadly-able full-agents with hidden intentions. Making this an especially odd time to complain that new AI systems might have killed us all.”

Seems not much has changed in the Yudkowsky vs. Hanson position over the years, i.e. still assigning high vs. low existential risk.
# Switching to Electric Mandolin

A little over a year ago I bought a [solid-body electric mandolin](https://goldtonemusicgroup.com/goldtone/instruments/gme-4). At the time I was thinking I would still mostly play my [acoustic](https://www.jefftk.com/p/new-mandolin) or bring both to gigs, but instead I've ended up switching entirely. I [compared recordings](https://www.jefftk.com/p/acoustic-vs-electric-mandolin) and [amplification](https://www.jefftk.com/p/microphone-on-electric-mandolin) [options](https://www.jefftk.com/p/microphone-on-electric-mandolin-ii), and it turns out that the cleaner sound works much better in the chaos of a dance hall. I also really like that it works [with a talk box](https://www.jefftk.com/p/playing-with-a-talk-box) and that I don't need to worry about feedback on stage anymore.

[![](https://www.jefftk.com/anna-playing-electric-mandolin.jpg)](https://www.jefftk.com/anna-playing-electric-mandolin-big.jpg)

I still play the acoustic when I'm jamming with friends or family, but it now spends most months without getting out of its case. Which does make me sad; it's a good mandolin and I enjoy playing it.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02bcigScJHbzCsSkKwpsgwnqNy1ABk2r4Nyur6Bq8m2aoCbpVC2VSeeJYNw18kD76Ll)*
# Does polyamory at a workplace turn nepotism up to eleven?

I am reading debates on the Effective Altruism forum, and they seem to focus on how a low-status person might be afraid to refuse to consent to sex with a high-status person, because the high-status person might then refuse to provide them a grant or a job position.

True, but this seems to be just the tip of the iceberg.

Making a high-status *polyamorous* person angry potentially means not just creating one enemy, but rather a *group* of enemies -- namely the person's entire polycule. They are naturally likely to *trust* that person. Even if they have doubts, they still have a selfish incentive not to *disrupt* their own polycule. And if they often meet and talk in private, the person will be able to tell all of them their version *first*.

*

Imagine the usual monogamous workplace. Imagine rejecting the advances of your manager, and then experiencing retaliation. Imagine contacting HR to complain, and finding out that the HR person handling your complaint is the manager's wife. Would you expect a fair treatment?

Now imagine a workplace with lots of polyamorous dating. The difference is that if your manager is polyamorous, the chance of meeting a member of their polycule is *higher*, and you probably *wouldn't even know*. Or you might give up and try your luck at a *different* company in this tightly connected small industry... but the *new* manager, or the job interviewer might be a part of the same polycule, too. Perhaps you have just locked yourself out of the entire industry.

On the other hand, joining a polycule of high-status people might be the ultimate *office politics* move. (Unless you had a bad luck, and the polycule turns out to be *two-tiered*: the high-status members who treat each other seriously, and the "underlings with benefits" that keep coming and going.)

*

This is not about "polyphobia".

I am not saying that monogamy is good and polyamory is bad. The failures of monogamy are numerous and *well known*, and we have lots of *rules* created around them. The rules against nepotism were created in the context of monogamy. The rules against sexual harassment were created in the context of (at least officially) monogamy. The archetype of a vindictive rejected man (or a woman) taking a revenge also exists mostly in the context of monogamy. We already know that monogamy can be bad, and we have a set of rules trying to minimize the harm (arguably, not sufficiently, but better than nothing).

I am just assuming that if members of the same species start dating polyamorously, we should expect some of that to happen there, too. And that perhaps we should adopt similar rules. I expect that most polyamorous people are nice and friendly, just like most monogamous people. I also expect that each group might have a few sexual predators.

The main difference is that polyamory seems like it could *scale up* some of the problems. (Larger group size. Possibility to hit on multiple people at the same time without violating a social norm.) The problematic part is "at the workplace"... but polyamory could be a multiplier of the problems.
# Halifax Monthly Meetup: AI Safety Discussion

In the interest of growing the LW/EA/ACX community in Halifax, we are hosting monthly meetups. Newcomers are especially welcome!

This month we'll be discussing AI safety. Why might AI alignment be hard? How fast might AI takeoff be? How can these be influenced? 

**All this and more at Seven Bayes!** 

Location: Seven Bays Cafe (2017 Gottingen Street) – [87PRMC29+9C](https://plus.codes/87PRMC29+9C)

Contact: [conorbarnes93@gmail.com](mailto:conorbarnes93@gmail.com)
# Coordination explosion before intelligence explosion...?

*Epistemic status: musing that I wanted to throw out there.*

A traditional AI risk worry has been the notion of an intelligence explosion: An AI system will rapidly grow in intelligence and become able to make huge changes using small[^cmtyqbwj7c8] subtle[^j2y85cnthk] tricks such as bioengineering or hacking. Since small actions are not that tightly regulated, these huge changes would be made in a relatively unregulated way, probably destroying a lot of things, maybe even the entire world or human civilization.

Modern AI systems such as LLMs seem to be making rapid progress in turning sensory data into useful information, aggregating information from messy sources, processing information in commonsense ways, and delivering information to people. These abilities do not seem likely to generalize to bioengineering or hacking (which involve generating novel capabilities), but they do seem plausibly useful for some things.

* * *

Two scenarios of interest:

**Coordination implosion:** Some people suggest that because modern AI systems are extremely error-prone, they will not be useful, except for stuff like spam, which degrades our coordination abilities. I'm not sure this scenario is realistic because there seem to be a lot of people working on making it work for useful stuff.

**Coordination explosion:** By being able to automatically do basic information processing, it seems like we might be able to do better coordination. We are already seeing this with chatbots that work as assistants, sometimes being able to give useful advice based on their mountains of integrated knowledge. But we could imagine going further, e.g. by automatically registering people's experiences and actions, and aggregating this information and routing it to relevant places.

(For instance, maybe a software company installs AI-based surveillance, and this surveillance notices when developers encounter bugs, and takes note of how they solve the bugs so that it can advise future developers who encounter similar bugs about what to do.)

This might revolutionize the way we act. Rather than having to create, spread, and collect information, maybe we would end up always having relevant information at hand, ready for our decisions. With a bit of rationing, we might even be able to keep spam down to a workable level.

* * *

I'm not particularly sure this is what things are going to look like. However I think the possibility is useful to keep in mind: There may be an intermediate phase between "full AGI" and now, where we have a sort of transformative artificial intelligence, but not in the sense of leading to an intelligence explosion. There may still be an intelligence explosion afterwards. Or not, if you don't believe in intelligence explosions.

I foresee privacy to be one counteracting force. These sorts of systems seem like they work better when they invade your privacy more, so people will resist that.

[^cmtyqbwj7c8]: Small = Involving relatively minor changes in terms of e.g. matter manually moved. 

[^j2y85cnthk]: Subtle = Dependent on getting many "bits" right at a distance.
# Steven Pinker on ChatGPT and AGI (Feb 2023)

While I disagreed with a lot of [Robin Hanson's latest take on AI risk](https://www.lesswrong.com/posts/AL6DRuE8s4yLn3yBo/robin-hanson-s-latest-ai-risk-position-statement), I am glad he came out with an updated position. I think with everything that's happened in the past 6-12 months, it's a good time for public intellectuals and prominent people who have previously commented on AGI and AI risk to check in again and share their latest views.  
  
That got me curious if Steven Pinker had any recent statements. I found this article on the Harvard Gazette from last month (Feb 2023), which I couldn't find posted on LessWrong before:

Article link
------------

[Will ChatGPT supplant us as writers, thinkers?](https://news.harvard.edu/gazette/story/2023/02/will-chatgpt-replace-human-writers-pinker-weighs-in/)  
[Q&A with Steven Pinker](https://news.harvard.edu/gazette/story/2023/02/will-chatgpt-replace-human-writers-pinker-weighs-in/)  
by Alvin Powell  
Feb 14, 2023

Summary
-------

Here's a summary of the article that ChatGPT generated for me just now (bold mine):

> Steven Pinker, a psychology professor at Harvard, has commented on OpenAI’s ChatGPT, an artificial intelligence (AI) chatbot that can answer questions and write texts. He is impressed with the AI's abilities, but also highlights its flaws, such as a lack of common sense and factual errors. Pinker believes that ChatGPT has revealed how statistical patterns in large data sets can be used to generate intelligent-sounding text, even if it does not have understanding of the world. **He also believes that the development of artificial general intelligence is incoherent and not achievable**, and that current AI devices will always exceed humans in some challenges and not others. Pinker is not concerned about ChatGPT being used in the classroom, as its output is easy to unmask as it mashes up quotations and references that do not exist.

Note that while he comments on AGI being an incoherent idea, **he doesn't speak specifically about existential risk from AI misalignment. So it's not totally clear, but I think we can infer Pinker considers the risk very low, since he doesn't think AGI is possible in the first place.**
# What should we do about network-effect monopolies?

Many large companies today are software monopolies that give their product away for free to get monopoly status, then do [the](https://github.com/uBlockOrigin/uBlock-issues/issues/338#issuecomment-496009417) [most](http://www.fbpurity.com/news/important-news-facebooks-legal-team-have-told-me-i-am-banned-from-facebook-because-of-f-b-purity/) [horrible](https://www.theverge.com/2019/6/28/19154220/grubhub-seamless-fake-restaurant-domain-names-commission-fees) [things](https://www.nytimes.com/wirecutter/blog/amazon-counterfeit-fake-products/) [once](https://www.theverge.com/2020/6/16/21293419/hey-apple-rejection-ios-app-store-dhh-gangsters-antitrust) [they’ve](https://www.polemicdigital.com/google-amp-go-to-hell/) [won](https://thetechnoskeptic.com/yelp-extortion-starring-role/). ([Previously](https://www.benkuhn.net/skinner/), [previously](https://www.benkuhn.net/product/).) Can we do anything about this?

Unfortunately, “you’re the product” is a popular business model for a reason: businesses like Facebook would be really hard to support without them.

Facebook would be suicidal to charge its users money, because its entire selling point is that everyone uses it, and “everyone” _hates_ paying money. In the US, Facebook makes over $25 per person on ads ([source](https://www.statista.com/statistics/251328/facebooks-average-revenue-per-user-by-region/)). Can you imagine if instead of ads they tried to charge people $25 a year?

Even on the margin, anything that costs Facebook users also makes it less valuable for its remaining users—it’s a negative feedback loop. The same goes for any other site where users create value for other users, like Twitter or Craigslist or Yelp or Wikipedia. (It’s not an accident that these are some of the most stagnant popular websites!)

In fact, this is a fundamental problem with [network effects](https://en.wikipedia.org/wiki/Network_effect) and low marginal costs. If a company wants to maintain a network effect, they need as many users as possible. If their marginal cost is low, then the easiest way to get users is to give the product away. To do that, they have to get paid by someone else. And when they start getting paid by someone else, they’ll inevitably start prioritizing that person’s interests.

Historically with other network-effect businesses, we’ve addressed this in a few different ways:

*   regulation (e.g. local utilities)
    
*   breakups (e.g. Bell)
    
*   standardization and interoperability (e.g. email, the Web, cryptocurrency)
    

So far for tech monopolies, people seem to be focused mostly on breakups—e.g. Facebook from Instagram/Whatsapp—but standardization seems to have produced much better outcomes in the past. (I like email and the Web a lot more than National Grid…) I’d be interested to see more exploration of that option!