Harley-ml/lesswrong huggingface dataset:
# Public Call for Interest in Mathematical Alignment

**Bottom line up front:** 

If you are currently working on, or are interested working in any area of mathematical AI alignment, we are collecting names and basic contact information to find who to talk to about opportunities in these areas. [If that describes you, please fill out the form](https://airtable.com/appKOZCNAGyp4qPqW/shrpiOcvBY17sY6lH)! (Please do so even if you think I already know who you are, or people will be left out!)

**More information**

There are several concrete research agendas in mathematical AI alignment, receiving varying degrees of ongoing attention, with relevance to different possible strategies for AI alignment. These include MIRI’s agent foundations and related work, [Learning Theoretic Alignment](https://www.alignmentforum.org/posts/ZwshvqiqCvXPsZEct/the-learning-theoretic-agenda-status-2023), [Developmental Interpretability](https://www.alignmentforum.org/s/SfFQE8DXbgkjk62JK/p/TjaeCWvLZtEDAS5Ex), Paul Christiano’s theoretical work, RL theory related work done at Far.AI, FOCAL at CMU, Davidad’s “Open Agency” architecture, as well as other work. Currently, as in the past, work in these areas has been conducted mainly in non-academic settings, often not published, and the people involved are scattered - as are other people who want to work on this research.

A group of people, including some individuals at MIRI, Timaeus, MATS, ALTER, PIBBSS, and elsewhere, are hoping to both promote research in these areas, and build bridges between academic and existing independent research. To that end, we are hoping to promote academic conferences, hold or sponsor attendance at research seminars, and announce opportunities and openings for PhD students or postdocs, non-academic positions doing alignment research, and similar. 

As a first step, we want to compile a list of people who are (at least tentatively) interested, and would be happy to hear about projects. This list will not be public, and is likely to involve very few emails to this list, but will be used to find individuals who might want to be invited to programs or opportunities.

Note that we are interested in people at all levels of seniority, including graduate students,  independent researchers, professors, research groups, university department contacts, and others who wish to be informed about future opportunities and programs.

**Interested in collaborating?**

If you are an academic, or are otherwise more specifically interested in building bridges to academia or collaborating with people in these areas, please mention that in the notes, and we are happy to be in touch with you, or help you contact others working in more narrow areas you are interested in.
# Boston Secular Solstice: Call for Singers and Musicans

As in past years the Boston EA/LW community is putting together a secular solstice celebration. It's a bit of a strange thing, somewhat like an atheist church service, with lots of group singing on silly and serious topics. For some of the flavor, see my [2022](https://www.jefftk.com/p/boston-solstice-2022-retrospective), [2019](https://www.jefftk.com/p/boston-solstice-2019-retrospective), and [2018](https://www.jefftk.com/p/boston-solstice-2018-retrospective) retrospectives. This year it's a bit late: 2023-12-30. More details on the [FB event](https://www.facebook.com/events/238479572316403).

This year I'm organizing the music again, and I'm looking for volunteers to lead songs and play instruments. For song leaders, we have a range of songs of varying difficulty, so as long as you can carry a tune and are comfortable standing up in front of a few dozen people there's probably something that would be a good fit for you. For musicians, I'd be enthusiastic about help both from people who want to play piano/guitar and accompany (especially if you read music and are excited about something like [this](https://secularsolstice.github.io/songs/Voicing_of_Fear/gen/Voicing-of-Fear-sheet-music.pdf)) and people who play violin/flute/cello etc and want to support the melody or play harmonies.

[![](https://www.jefftk.com/boston-solstice-2019-candles.jpg)](https://www.jefftk.com/boston-solstice-2019-candles-big.jpg)

If this sounds like a good time, let me know!
# What did you change your mind about in the last year?

Seems like a good New Year activity for rationalists, so I thought I'd post it early instead of late.

Here are some steps I recommend:

*   Go looking through old writings and messages. 
    *   If you keep a journal, go look at a few entries from throughout the last year or two.
    *   Skim over your LessWrong activity from the last year.
    *   If you're active on a slack or discord, do a search of `from: [your username]` and skim your old messages from the last year (or the last 3 months on the free tier of slack).
    *   Sample a few of your reddit comments and tweets from the last year.
    *   Same for text messages.
*   Think back through major life events (if you had any this year) and see if they changed your mind about anything. Maybe you changed jobs or turned down an offer; maybe you tried a new therapeutic intervention or recreational drug; maybe you finally told your family something important; maybe you finally excommunicated someone from your life; maybe you tried mediating a conflict between your friends.
*   Obvious, but look over your records of Manifold trades, quantitative predictions, and explicit bets. See if there's anything interesting.

Here are some emotional loadings that I anticipate seeing:

*   "Well, that aged poorly."
*   "Wow, bullet dodged!"
*   "But I mean...how could I have known?"
*   "Ah. Model uncertainty strikes again."
*   "Yeah ok, but *this* year it'll happen for sure!"
*   "\[Sigh\] I did have an inkling, but I guess I just didn't want to admit it at the time."
*   "I tested that hypothesis and got a result."
*   "Okay, they were right about this one thing. Doesn't mean I have to like them."
*   "Now I see what people mean when they say X"
*   "This is huge, why does no one talk about this?!"
# First and Last Questions for GPT-5*

Suppose that in the very near future, a research group finds that their conversational AI has begun to produce extremely high-quality answers to questions. There's no obvious limit to its ability, but there's also no guarantee of correctness or good intention, given the opacity of how it works. 

What questions should they ask it?

And what questions should they *not* ask it?
# What's the evidence that LLMs will scale up efficiently beyond GPT4? i.e. couldn't GPT5, etc., be very inefficient?

A lot of this recent talk about OpenAI, various events, their future path, etc., seems to make an assumption that further scaling beyond GPT4 will pose some sort of 'danger' that scales up linearly or super-linearly with the amount of compute. And thus pose an extraordinary danger if you plug in 100x and so on.

Which doesn't seem obvious at all.

It seems quite possible that GPT5, and further improvements, will be very inefficient.

For example, a GPT5 that requires 10x more compute but is only moderately better. A GPT6 that requires 10x more compute than GPT5, i.e. 100x more than GPT4, but is again only moderately better.

In this case there doesn't seem to be any serious dangers at all for LLMs. 

The problem self-extinguishes based on the fact that random people won't be able to acquire that amount of compute in the for-seeable future. Only serious governments, institutions, and companies, with multi-billion dollar cap-ex budgets will even be able to consider acquiring something much better. 

And although such organizations can't be considered to be perfectly responsible, they will still very likely be responsible enough to handle LLMs that are only a few times more intelligent.
# Goodhart's Law Example: Training Verifiers to Solve Math Word Problems

Sharing because of how clearly this paper demonstrates the risks associated with considering too many different solutions:  
  
"At test time, we can choose to generate arbitrarily many solutions to be judged by the verifier before selecting the highest-ranked completion. Figure 7a shows how 6B verifier performance varies with the number of completions per test problem. At this scale, performance improves as we increase the number of completions up to 400. Beyond this point, performance starts to decrease. This suggests that the benefits of search are eventually outweighed by the risk of finding adversarial solutions that fool the verifier. In general, we evaluate verifier test performance using 100 completions, since this captures most of the benefits of verification with a relatively modest compute cost."  
  
They propose and evaluate a solution to this:  
  
"To further increase performance, we can take a majority vote among the top verifier-ranked solutions instead of selecting only the single top solution. This voting process considers only the final answer reached by the individual solutions: the final answer selected is the one with the most votes. Figure 7b shows how performance varies as we allow a greater number of top samples to cast a vote. Unsurprisingly, when starting with a greater number of samples, we can afford to allow a greater number of samples to cast a vote. When we have only 100 samples, it is optimal to allow only the top 3-5 samples to cast a vote. When we have 3200 samples, it is approximately optimal to allow the top 30 to cast a vote."

Further empirical investigation into Goodhart's Law could prove valuable for alignment.
# The two paragraph argument for AI risk

The very short version of the AI risk argument is that an AI that is \*better than people at achieving arbitrary goals in the real world\* would be a very scary thing, because whatever the AI tried to do would then actually happen. As stories of magically granted wishes and sci-fi dystopias point out, it's really hard to specify a goal that can't backfire, and current techniques for training neural networks are generally terrible at specifying goals precisely. If having a wish granted by a genie is dangerous, having a wish granted by a genie that can't hear you clearly is even more dangerous.

Current AI systems certainly fall far short of being able to achieve arbitrary goals in the real world better than people, but there's nothing in physics or mathematics that says such an AI is \*impossible\*, and progress in AI often takes people by surprise. People just don't know what the actual time limit is, and unless we humans have a good plan \*before\* someone makes a scary AI that has a goal, things are going go very badly.

* * *

I've posted versions of this two-paragraph argument in various places online and used it in person, and it usually goes over pretty well; I think it explains pretty clearly and simply what the AI x-risk community is actually afraid of. I figured I'd post it here for everyone's convenience.
# Announcing New Beginner-friendly Book on AI Safety and Risk

Concisely,  
I've just released the book [*Uncontrollable: The Threat of Artificial Superintelligence and the Race to Save the World*](https://www.amazon.com/Uncontrollable-Threat-Artificial-Superintelligence-World/dp/B0CNNYKVH1/ref=cm_cr_arp_d_product_top?ie=UTF8&fbclid=IwAR0A0Aoq10TT7lQExjC0SOmew34zbCLcJSed2kLpzz53JpwwGR48aKbf9Oc)

*   It's an engaging introduction to the main issues and arguments about AI safety and risk. Clarity and accessibility were prioritized. There are blurbs of support from Max Tegmark, Will MacAskill, Roman Yampolskiy and others.  
*   Main argument is that AI capabilities are increasing rapidly, we may not be able to fully align or control advanced AI systems, which creates risk. There is great uncertainty, so we should be prudent and act now to ensure AI is developed safely. It tries to be hopeful.
*   Why does it exist?   
    There are lots of useful posts, blogs, podcasts, and articles on AI safety, but there was no up-to-date book entirely dedicated to the AI safety issue that is written for those without any exposure to the issue. (Including those with no science background.)
*   This book is meant to fill that gap and could be useful outreach or introductory materials. 
*   If you have already been following the AI safety issue, there likely isn't a lot that is new for you. So, this might be best seen as something useful for friends, relatives, some policy makers, or others just learning about the issue. (although, you may still like the framing)
*   It's available on numerous Amazon marketplaces. Audiobook (edit) now available and Hardcover options to follow. 
*   It was a hard journey. I hope it is of value to the community.
# Corrigibility or DWIM is an attractive primary goal for AGI

While rereading the [List of Lethalities](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities) (LoL), I was compelled by the argument against corrigibility. It's really hard to make a goal of "maximize X, except if someone tells you to shut down". I think the same argument applies to Christiano's goal of achieving corrigibility through RL by rewarding correlates of corrigibility. If other things are rewarded more reliably, you may not get your AGI to shut down when you need it to.

But those arguments don't apply if [corrigibility in the broad sense](https://ai-alignment.com/corrigibility-3039e668638) is the primary goal. "Doing what this guy means by what he says" is a perfectly coherent goal. And it's a highly attractive one, for a few reasons. Perhaps corrigibility shouldn't be used in this sense and do what I mean (DWIM) is a better term. But it's closely related. It accomplishes corrigibility, and has other advantages. I think it's fairly likely to be the first goal someone actually gives an AGI.

"Do what I mean" sidesteps the difficulty of outer alignment. The difficulty of outer alignment is another point in the LoL. One common plan, which seems sensible, is to keep humans in the loop; to have a [Long Reflection](https://forum.effectivealtruism.org/topics/long-reflection) to decide what we want. "DWIM" allows you to contemplate and change your mind as much as you like.

Of course, the problem here is: do what WHO means? We'd like an AGI that serves all of humanity, not just one guy or board of directors. And we'd like to not have power struggles.

But from the point of view of a team actually deciding what goal to give their shot at AGI, DWIM will be incredibly attractive for practical reasons. The outer alignment problem is hard. Specifying one person (or a few) to take instructions from is vastly simpler than deciding and specifying a goal that captures all of human flourishing for all time. You don't want to trust an AGI to interpret that goal correctly. Intepreting DWIM is still fraught, but it is naturally self-correcting, and becomes more useful as the AGI gets more capable. A smarter AGI will be better at understanding what you probably mean, and better at realizing when it's not sure what you mean so it can ask for clarification.

This doesn't at all address inner alignment. But when somebody thinks they have good-enough inner alignment to launch a goal-directed, [sapient](https://www.lesswrong.com/posts/WqxGB77KyZgQNDoQY/sapience-understanding-and-agi) AGI, DWIM is likely to be the goal they'll choose. This could be good or bad, depending on how well they've implemented inner alignment, and what type of people they are.
# Paper out now on creatine and cognitive performance

Our paper “The effects of creatine supplementation on cognitive performance - a randomised controlled study” is out now!

→ Paper: https://doi.org/10.1186/s12916-023-03146-5

→ Twitter thread:  https://twitter.com/FabienneSand/status/1726196252747165718?t=qPUghyDGMUb0-FZK7CEXhw&s=19

Jan Brauner and I are very thankful to Paul Christiano for suggesting doing this study and for funding it.
# why did OpenAI employees sign

Recently, OpenAI employees signed [an open letter](https://www.nytimes.com/interactive/2023/11/20/technology/letter-to-the-open-ai-board.html) demanding that the board reinstate Sam Altman, add other board members (giving some names of people allied with Altman), and resign, or else they would quit and follow Altman to Microsoft.

Following those demands would've put the entire organization under the control of 1 person with no accountability to anyone. That doesn't seem like what OpenAI employees wanted to be the case, unless they're dumber than I thought. So, why did they sign? Here are some possible reasons that come to mind:

1. Altman is just really likeable for people like them - they just like him.
2. They felt a sense of injustice and outrage over the CEO being fired that they'd never felt over lower-level employees being fired.
3. They were hired or otherwise rewarded by Altman and thus loyal to him personally.
4. They believed Altman was more ideologically aligned with them than any likely replacement CEO (including Emmett Shear) would be.
5. They felt their profit shares would be worth more with Altman leading the company.
6. They were socially pressured by people with strong views from (3) or (4) or (5).
7. They were afraid the company would implode and they'd lose their job, and wanted the option of getting hired at a new group in Microsoft, and the risk of signing seemed low once enough other people already signed.
8. They were afraid Altman would return as CEO and fire or otherwise punish them if they hadn't signed.
9. Something else?

Which of those reasons do you think drove people signing that letter, and why do you think so?
# Could World War I have been prevented given the benefit of hindsight?

Consider the following scenario: you are sent back in time some number of years before the beginning of WWI, with the goal of preventing the war. This includes preventing similar wars that happen slightly earlier or slightly later - you are aiming for a peaceful, stable coexistence between the European powers. Unsophisticated strategies like warning The Archduke against going to Sarajevo may simply result in another similar casus belli happening. 

To prevent silly strategies, minor details of the world you are sent to will be changed, so dates and names may differ very slightly, but the overall WWI path will unfold almost exactly as it did in our timeline. 

Furthermore, you carry some evidence of extraordinary status that is credible to the leaders of WWI nations. They will grant you an audience, but they will not blindly trust you. You may lie to them or tell them the truth. 

You don't have detailed designs for post-1914 technology, but you have an ordinary informed layman's understanding of actual history. So you can tell them that nuclear bombs will be invented, but not exactly how, and they may or may not believe you. 

Please answer with your strategy for saving Europe from World War I, and why you think it would work (including in the face of likely reactions from the major players)!
# Could Germany have won World War I with high probability given the benefit of hindsight?

Related: [Could World War I have been prevented given the benefit of hindsight?](https://www.lesswrong.com/posts/PBNpadZKBGFf6Taa3/could-world-war-i-have-been-prevented-given-the-benefit-of)

Consider the following scenario: you are sent back in time some number of years before the beginning of WWI, say 1904 for concreteness, with the goal of changing Imperial German strategy before and during the war such that Germany wins with very high probability [meaning that throughout many small deviations of the timeline, including small variations in the enemy strategic response function, at least 95% of them lead to an Austro-German victory over at least France and Russia, taking lots of new territory and establishing a German empire on the world stage. We will define winning as either total destruction of the enemy military or a surrender by the Russia/France side of the war where Germany gets to dictate terms.]

This includes winning similar wars that happen slightly earlier or slightly later - you are aiming for a durable strategy for Germany and Austro-Hungary that makes them win over the opposing European powers. 

To prevent silly strategies, minor details of the world you are sent to will be changed, so for example dates and names may differ very slightly. 

Furthermore, you carry some evidence of extraordinary status that is credible to the leaders of Austro-Hungary and Germany. They will grant you an audience and they have an unusually large amount of patience to listen to you, but they will not blindly trust you. You need to convince them based on reasoning that they will accept. You may lie to them or tell them the truth. I will also stipulate that if you manage to convince the leadership of a plan, they will actually follow it with an unusually high degree of fanaticism and coherence. 

You don't have detailed designs for post-1914 technology, but you have an ordinary informed layman's understanding of actual history. So you can tell them that nuclear bombs will be invented, but not exactly how, and they may or may not believe you.

Please answer with your strategy and why you think it would work (including in the face of likely reactions from the major players)!
# How can I use AI without increasing AI-risk?

I barely use AI tools, and the main reason is that I developed some sort of revulsion to them, because associate them with the harms they cause and the risks they bring.

On the other hand, it seems that more and more, whoever doesn't adopt AI tools will get become far less productive and will be left behind.

And also, if the people who worry about the risk limit themselves, and the people who don't worry about the risk don't limit themselves, it creates a [personal responsibility vortex](https://www.youtube.com/watch?v=SjNRtrZjkfE&pp=ygUdcGVyc29uYWwgcmVzcG9uc2liaWxpdHkgdm90ZXg%3D) which tilts the balance of power away from safety.

But, perhaps it's possible to get the best of both words - use AI tools , but in a responsible manner that doesn't increase the harms and risks from it.

How does one do that?
# Update #2 to "Dominant Assurance Contract Platform": EnsureDone

This is the second update to [The Economics of the Asteroid Deflection Problem (Dominant Assurance Contracts)](https://www.lesswrong.com/posts/CwgHX9tbfASqxjpsc/the-economics-of-the-asteroid-deflection-problem-dominant). 

It took a bit longer than I expected but I finally launched a platform for raising money using Dominant Assurance Contracts: [EnsureDone](https://ensuredone.com).

Why is it called EnsureDone? Well, [we ran a naming contest on manifold.markets](https://manifold.markets/market/what-will-i-call-my-dominant-assura), and EnsureDone had the most [Keynesian Beauty](https://en.wikipedia.org/wiki/Keynesian_beauty_contest). 

We've launched with three projects

*   [Building a Platform and Organization to Foster Logic Tournaments Around the World](https://ensuredone.com/projects/2023-buridan) By Sebastian Garren **$1800.** Closing 2024-02-08
*   [Keep tabs on your local council](https://ensuredone.com/projects/2023-kingston) By Parth Pasnani **$550**. Closing 2023-12-30
*   [Reflective clouds in the stratosphere to cool the Earth](https://ensuredone.com/projects/2023-makesunsets) By Make Sunsets **$1000**. Closing 2023-12-26

If any of these projects sound interesting, remember that

*   If they don't raise enough money you get all your money back + a refund bonus
*   If they do raise enough money you will get the public good.

Pledging is a win-win situation.

If you are interested in producing public goods and using a Dominant Assurance Contract to raise funds [apply to have your project featured on EnsureDone!](https://docs.google.com/forms/d/e/1FAIpQLScN7fYrL4X_Zk2hxzk1oLNClzWz7wNBW1CY8PEyY8qqiRKc-g/viewform)

*What's next?*

Since I have a product I can sell to people. I'm going to focus the next few months on sales. 

I'm unlikely to post again on LessWrong, so if you are interested in this project [join our Discord.](https://discord.gg/QYfJaV4HXC)
# Thoughts on teletransportation with copies? 

I’m interested in how people here feel about the [teletransportation paradox](https://en.wikipedia.org/wiki/Teletransportation_paradox) with multiple copies. I have several scenarios: 

*Scenario 1:* You are placed in a machine on earth that puts you to sleep and then instantaneously disintegrate your body, in the process recording its exact atomic configuration. 

This information is then beamed to another machine on planet A, and in that machine new matter is used to construct a body with the same configuration as yours. 

At the same time, the same information is sent to a machine on planet B, and a second body is also constructed with the same configuration as yours, at the exact same time as the one on planet A. 

At this point, both copies of your atomic configuration wake up and walk out into their respective planets. 

***Question 1: What is the probability that after going to sleep on earth, you wake up on planet A***?

Now we modify the scenario:

*Scenario 2:* 1 millisecond after the initial split, and before either copy wakes up, the machine on planet B is activated again, the copy on planet B is instantaneously disintegrated, and that atomic configuration is transmitted to two more machines on planet C and planet D, which both construct identical copies of the original copy on planet B. 

At this point, the copies on planet A, C, and D are all woken up and walk out into their respective planets. 

***Question 2: What is the probability that, in this new scenario, after going to sleep on earth, you wake up on planet A***?

*Scenario 3&4: *The teletransporters are capable of copying not just you, but also your clothing and items you are holding, such as the cash and other contents your wallet. On the way to the first machine, someone offers to sell you a winning lottery ticket that pays out a hundred dollars, but only if cashed in on Planet A. They only take cash, and if you pay the cash, it obviously will not be copied in the teletransportation process. Assume that you are purely selfish about your own experiences.

***Question 3: In scenario 1, how much cash would you pay for the 100 buck planet A lottery ticket? ***

***Question 4: In scenario 2, how much cash would you pay for the 100 buck planet A lottery ticket?***

I have my own thoughts and confusions, but I’d be interested in what other people think first. Note that answers like "not enough information" or "ill formed question" are valid as well.
# The origins of the steam engine: An essay with interactive animated diagrams

This is a guest post written by Anton Howes and animated by Matt Brown of Extraordinary Facility. This project was sponsored by The Roots of Progress, with funding generously provided by The Institute:

> Steam power did not begin with the steam engine. Long before seventeenth-century scientists discovered the true nature of vacuums and atmospheric pressure, steam- and heat-using devices were being developed. Here we’ll explore the long, little-known story of how the steam engine evolved. And have fun playing with the ancient devices.

Some of the interactive animated diagrams:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/animations/73ee202c5a68cc7d85fac7492b83fe208b5a95d1de2e8aaf.gif)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/animations/16913de38ac75e949e41073bf98af29db31234de4409c17d.gif)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/animations/ab318c25bd3355d73b95537e822ae154a03d3516fa8e8df0.gif)

[Read the full essay and try out the diagrams here](https://rootsofprogress.org/steam-engine-origins).
# Losing Metaphors: Zip and Paste

In python (and several other languages) if I have two lists and want to process corresponding elements together I can use [`zip`](https://en.wikipedia.org/wiki/Zipping_(computer_science)):

>>\> for number, letter in zip(
...    \[1,2,3,4\], \["a", "b", "c", "d"\]):
...  print(number, letter)
...
1 a
2 b
3 c
4 d

The metaphor is a zipper, taking the two sides and merging them together. It's not perfect, since a zipper interleaves instead of matching pairs, but it's pretty good.

In unix, there's a command line tool, `paste` that does the same thing:

$ paste <(echo 1 2 3 4 | tr ' ' '\\n') \
        <(echo a b c d | tr ' ' '\\n')
1 a
2 b
3 c
4 d

This time the metaphor is pasting: physically putting one column next to another.

I found a [discussion](https://news.ycombinator.com/item?id=29846318) on the origin of these terms, which traces `paste` back to at least 1978 at Bell Labs Center 127. The earliest use of `zip` I've found is 1988's [Introduction to Functional Programming](https://usi-pl.github.io/doc/Bird_Wadler.%20Introduction%20to%20Functional%20Programming.1ed.pdf) (p57).

What I find interesting about these names is that they've both "lost" in a sense: `paste` and `zip` generally mean something else to computer users:

*   Pasting is now almost always used in the "[copy and paste](https://en.wikipedia.org/wiki/Cut,_copy,_and_paste)" sense of moving data through the computer's clipboard from one program to another.
    
*   Zipping is used to mean archiving, after the `PKZIP` tool. Which seems to have become a metaphor only retroactively: it's creators [called](http://www.bbsdocumentary.com/library/CONTROVERSY/LAWSUITS/SEA/) it "ZIP" as in "zippy", and the idea of zipping and unzipping an the way you would a bag seems to have come later. Perhaps a [retrometaphore](https://en.wikipedia.org/wiki/Backronym)?
    

Older names for this are less metaphorical: APL used the comma-operator, and in Lisp this [would be done](https://stackoverflow.com/questions/6195776/zipping-lists-together-in-common-lisp-problem-with-and) with the more general `mapcar` function.
# Inositol Non-Results

[Three months ago](https://acesounderglass.com/2023/09/08/luck-based-medicine-inositol/) I suggested people consider inositol for treating combined vague mood issues and vague stomach issues. I knew a few people who’d really benefited from it, and when one talked about it on his popular Twitter account several more people popped up thanking him for the suggestion, because it fixed their lives too. But those reports didn’t come with a denominator, which made it hard to estimate the success rate; I was hoping mentioning it on my blog and doing a formal follow-up to capture the non-responders would give a more accurate number.

Unfortunately, I didn’t get enough people to do anything useful. I received 7 responses, of which 3 didn’t have digestive issues and thus weren’t really the target. The low response rate might be a consequence of giving the wrong link in the original follow-up post, or maybe it just wasn’t that interesting. I’m reporting the results anyway out of a sense of civic virtue. 

Of those 4 remaining responses:

*   2 rated it exactly 5 out of 10 (neutral)
*   1 rated it as 6, which was not strong enough for them to try it a third time.
*   1 rated it as 3- not bad enough that they spontaneously noticed a problem, but they did detailed mood tracking and the linear regression was clearly bad. 

That response rate isn’t really low enough to prove anything except that anything with a real effect can hurt you, and the value of detailed data. So for now we just have David’s estimate that 5% of people he inspired to take inositol benefited from it.
# Stupid Question: Why am I getting consistently downvoted?

I feel like I've posted some good stuff in the past month, but the bits that I think are coolest have pretty consistently gotten very negative karma.

I just read the rude post about rationalist discourse basics, and, while I can guess why my posts are receiving negative karma, that would involve a truly large amount of speculating about the insides of other people's heads, which is apparently discouraged. So I figured I would ask.

I will offer a bounty of $1000 for the answer I find most helpful, and a bounty of $100 for the next most helpful three answers. This will probably be paid out over Venmo, if that is a decision-relevant factor.

Note that I may comment on your answer asking for clarification.

Edit 11-30-2023 1:27 AM: I have selected the recipients of the bounties. The grand prize of $1000 goes to [@Shankar Sivarajan](https://www.lesswrong.com/users/shankar-sivarajan?mention=user) . The three runner-up prizes of $100 go to [@tslarm](https://www.lesswrong.com/users/tslarm?mention=user) , [@Joe Kwon](https://www.lesswrong.com/users/joe-kwon?mention=user) , and [@trevor](https://www.lesswrong.com/users/trevor-one?mention=user) . Please respond to my DM to arrange payment or select a worthy charity to receive your winnings.

Edit 11-30-2023 12:08 PM: I have paid out all four bounties. Please contact me in DM if there is any issue with any of the bounties.
# Feature Request for LessWrong

Can we add posts with negative karma to a Sequence? I'm working on a Sequence for my alignment research agenda, and the most valuable content in the sequence is all currently underwater. I get that LessWrong wants the community to have veto authority over people creating Sequences and adding controversial posts to them, but I am not sure how to proceed with my alignment work in the absence of a quorum of people willing to upvote the most valuable and important parts of my research agenda.
# Researchers and writers can apply for proxy access to the GPT-3.5 base model (code-davinci-002)

AI safety researchers and creative writers can apply for proxied access to the GPT-3.5 base model through, also known as [code-davinci-002](https://cyborgism.wiki/hypha/code-davinci-002). 

Publicly available base models include "davinci-002" (the text-davinci-xxx models are not base models) and cohere's "xlarge" model.

I recommend interacting with language models using janus' loom. One such implementation of it by [parafactual](https://www.lesswrong.com/users/parafactual) is [Loomsidian](https://github.com/cosmicoptima/loom). In brief, the Loom is a tool for generating multiple continuations of a prompt and organizing them in a tree. The Loom babbles, and you prune its continuations. It allows for creative exploration of questions with multiple answers.
# Queuing theory: Benefits of operating at 60% capacity

Related to [Slack](https://www.lesswrong.com/s/xEkBK3SRrp6DhGf7c/p/yLLkWMDbC9ZNKbjDG). Related to [Lean Manufacturing](https://en.wikipedia.org/wiki/Lean_manufacturing), aka JIT Manufacturing.

TL;DR A successful task-based system should sometimes be idle, like [40% of worker ants.](https://www.sciencealert.com/many-worker-ants-are-actually-lazy-slackers-but-there-s-a-good-reason-for-that)

Doing tasks quickly is essential for producing value in many systems. For software teams, delivering a feature gives valuable insight into user needs, which can improve future feature quality. For supply chains, faster delivery releases capital for reinvestment. However, the relationship between capacity utilized and service time is exponential, as shown by the diagram below.

![queueing-6.png](https://less.works/img/queueing_theory/xqueueing-6.png.pagespeed.ic.zBb72ZL5e3.png)

Stochastic arrivals mean it's better to sometimes have idle capacity, so that the system is quickly ready for when tasks arrive in droves.

A heuristic we can derive from queuing theory is that the optimal balance between efficiency and capacity typically occurs when the system is around 30-40% idle. For a single producer system, being X% idle is that producer being idle X% of the time. For a multi-producer system, being X% idle is X% of those producers being idle on average. This heuristic applies best to systems involving lots of discrete, oddly-shaped tasks.

The linked post explains this theory in more detail, and gives examples of where queues appear in the real world. See also the [Wikipedia article](https://en.wikipedia.org/wiki/Queueing_theory):

> **Queueing theory** is the mathematical study of waiting lines, or [queues](https://en.wiktionary.org/wiki/queue).[^\[1\]^](https://en.wikipedia.org/wiki/Queueing_theory#cite_note-sun-1) A queueing model is constructed so that queue lengths and waiting time can be predicted.[^\[1\]^](https://en.wikipedia.org/wiki/Queueing_theory#cite_note-sun-1) Queueing theory is generally considered a branch of [operations research](https://en.wikipedia.org/wiki/Operations_research) because the results are often used when making business decisions about the resources needed to provide a service.
# Could there be "natural impact regularization" or "impact regularization by default"?

Specifically, imagine you use [general-purpose](https://www.lesswrong.com/posts/6mysMAqvo9giHC4iX/what-s-general-purpose-search-and-why-might-we-expect-to-see) search procedure which recursively invokes itself to solve subgoals for the purpose of solving some bigger goal.

If the search procedure's solutions to subgoals "change things too much", then they're probably not going to be useful. E.g. for Rubik's cubes, if you want to swap some of the cuboids, it does you know good if those swaps leave the rest of the cube scrambled.

Thus, to some extent, powerful capabilities would have to rely on some sort of impact regularization.

I'm thinking that natural impact regularization is related to the notion of "elegance" in engineering. Like if you have some bloated tool to solve a problem, then even if it's not strictly speaking an issue because you can afford the resources, it might feel ugly because it's excessive and puts mild constaints on your other underconstrained decisions, and so on. Meanwhile a simple, minimal solution often doesn't have this.

Natural impact regularization wouldn't *guarantee* safety, since it's still allows deviations that don't interfere with the AI's function, but it sort of reduces one source of danger which I had been thinking about lately, namely I had been thinking that the instrumental incentive is to search for powerful methods of influencing the world, where "power" connotes the sort of raw power that unstopably forces a lot of change, but really the instrumental incentive is often to search for "precise" methods of influencing the world, where one can push in a lot of information to effect narrow change.[^csrxgqulp65]

Maybe another word for it would be "natural inner alignment", since in a sense the point is that capabilities inevitably select for inner alignment. Here I mean "natural" in the sense of natural abstractions, i.e. something that a wide variety of cognitive algorithms would gravitate towards.

[^csrxgqulp65]: A complication is that any one agent can only have so much bandwidth, which would sometimes incentivize more blunt control. I've been thinking bandwidth is probably going to become a huge area of agent foundations, and that it's been underexplored so far. (Perhaps because everyone working in alignment sucks at managing their bandwidth? 😅)
# South Bay Pre-Holiday Gathering

Ilmarin House is hosting its third unstructured holiday get-together! There'll be cookies and rudimentary snacks; apart from that it's a potluck and you should bring what (if anything) you'd like to have available. 

Invite some friends if you think they'd enjoy the event as well! An RSVP would be appreciated.
# 2023 Unofficial LessWrong Census/Survey

~~**The Less Wrong General Census is unofficially here! You can take it at**~~ [~~**this link.**~~](https://docs.google.com/forms/d/e/1FAIpQLSc_-Te1a4Q-FuNhwwgSGDwVWCHOWppCrWtKaVrhHcujopy5Lg/viewform?usp=sf_link)

Update: The census is closed! Thank you everyone who took it.

It's that time again.

If you are reading this post and identify as a LessWronger, then you are the target audience. I'd appreciate it if you took the survey. If you post, if you comment, if you lurk, if you don't actually read the site that much but you do read a bunch of the other rationalist blogs or you're really into HPMOR, if you hung out on rationalist tumblr back in the day, or if none of those exactly fit you but I'm maybe getting close, I think you count and I'd appreciate it if you took the survey.

Don't feel like you have to answer all of the questions just because you started taking it. Last year I asked if people thought the survey was too long, collectively they thought it was maybe a little bit too long, and then I added more questions than I removed. The survey is structured so the fastest and most generally applicable questions are (generally speaking) towards the start. At any point you can scroll to the bottom and hit Submit, though you won't be able to change your answers once you do.

The questions are a mix of historical questions that were previously asked on the LW Census, new questions sourced from LW commenters and some rationalist adjacent organizations I reached out to, and the things I'm curious about. This includes questions from a list a member of the LessWrong team sent me when I asked about running the census.

The survey shall remain open from now until at least January 1st, 2024. I plan to close it sometime on Jan 2nd. 

I don't work for LessWrong, and as far as I know the LessWrong Census organizer has never been someone who worked for LessWrong. Once the survey is closed, I plan to play around with the data and write up an analysis post [like this one](https://www.lesswrong.com/posts/E2koS25u6jajt5uo8/2022-survey-results). 

~~**Remember, you can take the survey at**~~ [~~**this link**~~](https://docs.google.com/forms/d/e/1FAIpQLSc_-Te1a4Q-FuNhwwgSGDwVWCHOWppCrWtKaVrhHcujopy5Lg/viewform?usp=sf_link)~~**.**~~ **Update: The census is closed, thank you everybody who took it! You can see the analysis post** [**here**](https://www.lesswrong.com/posts/WRaq4SzxhunLoFKCs/2023-survey-results)**.**

Once upon a time, there was a tradition that if you took the survey you could comment here saying you had done so, and people would upvote you and you would get karma.
# Disappointing Table Refinishing

Shaving cream on a table: good kids activity on a wet day, right?

[![](https://www.jefftk.com/table-with-shaving-cream.jpg)](https://www.jefftk.com/table-with-shaving-cream-big.jpg)

Unfortunately it turns out our table's finish is alcohol-soluble, and spending a couple hours covered with a high-alcohol foam was not good for it:

[![](https://www.jefftk.com/definished-table.jpg)](https://www.jefftk.com/definished-table-big.jpg)

The center leaf I made, and is polyurethane, but we decided the original portions were probably shellac.

Initially, we put plastic sheeting on it (always have extra plastic sheeting on hand!) to protect it from further damage:

[![](https://www.jefftk.com/plastic-covered-table.jpg)](https://www.jefftk.com/plastic-covered-table-big.jpg)

I looked up how to shellac tables, bought flakes and alcohol, and mixed up some 1.5lb. I sanded the damaged finish to get it smooth, but wasn't trying to fully remove it. I did ~5 layers of shellac, letting it dry overnight between them and covering with plastic during the day so the table would still be usable.

[![](https://www.jefftk.com/reshellaced-table.jpg)](https://www.jefftk.com/reshellaced-table-big.jpg)

Unfortunately, it didn't really work: each evening when I went to put on another layer a few spots of finish would come up with the plastic. And even over a month later it's not completely cured: when we put a hot dinner dish on a cork trivet, at the end of the meal there will be little bits of cork stuck to the table. It's also way less resilient to scraping and other damage than the original finish was.

[![](https://www.jefftk.com/table-with-sad-finish.jpg)](https://www.jefftk.com/table-with-sad-finish-big.jpg)

At this point we're just ignoring it, but someday I'd like to fix it properly. Possibly I'll strip the whole surface and put on polyurethane? Advice?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0rGy1ArUMbGKY1SPJn2cC2hHLNVJQi5yGgG3T7vkwCJavwcsuqL1EQDFfdSmFtck1l), [mastodon](https://mastodon.mit.edu/@jefftk/111519860017538285)*
# n of m ring signatures

A normal cryptographic signature associated with a message and a public key lets you prove to the world that it was made by someone with access to the private key associated with the known public key, without revealing that private key. You can read about it on Wikipedia [here](https://en.wikipedia.org/wiki/Digital_signature).

A ring signature associated with a message and a set of public keys lets you prove to the world that it was made by someone with access to the message and one private key associated to one of the public keys in the set, but nobody will be able to tell which public key it was. This lets you say something semi-anonymously, which is neat. It’s also used in the private cryptocurrency Monero. You can read about them on Wikipedia [here](https://en.wikipedia.org/wiki/Ring_signature).

Here’s a thing that would be better than a ring signature: a signature that proved that it was made by a subset of public keys of a certain size. In my head, I was calling this an n of m ring signature for a while. But when I googled “n of m ring signature”, nothing came up. It turns out this is because in the literature, it’s called a “threshold ring signature”, a “k of n ring signature”, or a “t of n ring signature” instead. I think perhaps the first paper about it is [this one](https://www.iacr.org/archive/crypto2002/24420467/24420467.pdf), but I haven’t checked very hard.

Anyway: I would like to make it so that when you search for n-of-m ring signatures online, you find a thing telling you that you should instead search for “threshold ring signature”. Hence this post.
# Open Thread – Winter 2023/2024

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).