Harley-ml/lesswrong huggingface dataset:
# What's next for instrumental rationality?

Preceded by: [Curating "The Epistemic Sequences" (list v.0.1)](https://www.lesswrong.com/posts/kS2A5wvMEuu5SjXj9/curating-the-epistemic-sequences-list-v-0-1)

*Epistemic status: speculations and ideations from me about the potential for further progress on broadly accessible instrumental rationality content.*

In [Curating "The Epistemic Sequences"](https://www.lesswrong.com/posts/kS2A5wvMEuu5SjXj9/curating-the-epistemic-sequences-list-v-0-1), I explained how the epistemic content of the LessWrong sequences has a different epistemic status than the instrumental content, i.e.., content on how to behave.  So what's next for instrumental rationality?  It would be great if there were a how-to-behave version of the sequences that was built on foundations as strong as logic, probability, statistics, and causal inference.  

Unfortunately, those foundations don't yet exist.  There aren't formal foundations for decision theory, game theory, ethics, meta-ethics, and political theory that are "tried and true" the way logic, probability, statistics, and causal inference ("L+P+S+C") are.

Many people argue that universally-useful how-to-behave instructions *can't* exist, on the the grounds that "philosophers have been trying to solve these issues for millennia", but I'm not sure that's a strong case. After all, philosophers had been trying to develop truth-seeking techniques for millennia prior to the 20th century, and then along came a bunch of progress in L+P+S+C with widespread applications, enabling what might be called an "epistemic enlightenment (for individuals)", which arguably culminated in the epistemic content of the LessWrong sequences.  And, perhaps in the next decade, there could be progress in the theory of [embedded agency](https://www.lesswrong.com/tag/embedded-agency) and multi-agent rationality ("E+M") leading to real-world applications as robustly useful and well-vetted as L+P+S+C are today.  If breakthroughs in embedded & multi-agent rationality then remained in practice for something like 30 years of applications in broad-sweeping domains (or, an AI-augmented equivalent of 30 human-civilization-years) the way L+P+S+C have, perhaps then will be a good time for someone to write the "The Instrumental Sequences", and a new generation of instrumentally enlightened people will look back and wonder why it was considered impossible to derive a principled account of how individuals should behave.
# Hiring Programmers in Academia

A professor I know through the EA community has been trying to hire a software engineer for their research, and they explained some privately about how this is tricky. The following are points I took away that might be useful to programmers considering job postings, people in academia looking to hire programmers, and people trying to understand why it's [hard](https://www.jefftk.com/p/simultaneous-shortage-and-oversupply) for EA projects to hire programmers despite there being a lot of them in EA.

*   Market rates for good programmers are much higher than universities are used to paying for anything. For example, a Senior Software Engineer at Google earns [~$350k](https://www.levels.fyi/?compare=Google&track=Software%20Engineer), while the most this professor's university would be willing to pay, full-time, would be ~$100k.
    
*   That the money is coming from a grant doesn't resolve this: the university would still not let you pay a higher salary because you need to go through university HR and follow their approach to compensation.
    
*   Universities will let you pay more when doing temporary arrangements, for example, as an employee on a six-month term or as an independent contractor, because you're no longer implicitly paying partly in job security. This might allow them to get up to $180k-$270k without benefits, though still below market rates.
    
*   Depending on university rules, if you did want to hire someone full-time you might have to pick from a pool of internal candidates, even if those candidates weren't very good, or else make a strong case for why no internal candidate had the required skills. Since programmers earning university wages could be earning maybe three times as much if they were good enough to get hired in industry and wanted to switch, many of them are not great.
    
*   They showed me a public job description that had a very long list of required skills, and seemed to be targeting a more experienced person than the work called for. Not knowing the situation, I initially thought that this was due to a misunderstanding about what kind of candidate they actually needed, but it's how they convinced the university that they should be able to offer this high an hourly rate.
    
*   While this researcher hasn't done this, others have bypassed these restrictions by hiring someone for "40hr/wk" with an understanding that they're actually going to be working much less than that.
    

One effect of this is that if an EA academic project needs a full-time experienced software engineer, they will often need to hire an EA because they need someone who is willing to work for much less than they could get elsewhere. Even if the work seems like it could be done by a non-EA at market wages, academically affiliated projects won't be able to do that.

(The professor reviewed this post, and has asked to remain anonymous.)
# Impact of " 'Let's think step by step' is all you need"?

*Title from:* [*https://twitter.com/arankomatsuzaki/status/1529278581884432385*](https://twitter.com/arankomatsuzaki/status/1529278581884432385)*.*

This is not quite a linkpost for [this](https://arxiv.org/abs/2205.11916) paper.

Nonetheless, the abstract is:

> Pretrained large language models (LLMs) are widely used in many sub-fields of natural language processing (NLP) and generally known as excellent few-shot learners with task-specific exemplars. Notably, chain of thought (CoT) prompting, a recent technique for eliciting complex multi-step reasoning through step-by-step answer examples, achieved the state-of-the-art performances in arithmetics and symbolic reasoning, difficult system-2 tasks that do not follow the standard scaling laws for LLMs. While these successes are often attributed to LLMs' ability for few-shot learning, we show that LLMs are decent zero-shot reasoners by simply adding ``Let's think step by step'' before each answer. Experimental results demonstrate that our Zero-shot-CoT, using the same single prompt template, significantly outperforms zero-shot LLM performances on diverse benchmark reasoning tasks including arithmetics (MultiArith, GSM8K, AQUA-RAT, SVAMP), symbolic reasoning (Last Letter, Coin Flip), and other logical reasoning tasks (Date Understanding, Tracking Shuffled Objects), without any hand-crafted few-shot examples, e.g. increasing the accuracy on MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% with an off-the-shelf 175B parameter model. The versatility of this single prompt across very diverse reasoning tasks hints at untapped and understudied fundamental zero-shot capabilities of LLMs, suggesting high-level, multi-task broad cognitive capabilities may be extracted through simple prompting. We hope our work not only serves as the minimal strongest zero-shot baseline for the challenging reasoning benchmarks, but also highlights the importance of carefully exploring and analyzing the enormous zero-shot knowledge hidden inside LLMs before crafting finetuning datasets or few-shot exemplars.

Consider the following image, taken from the paper:

![Image](https://pbs.twimg.com/media/FTkWofoVsAAnqu2?format=png&name=900x900)

According to the paper linked above, this technique of propmpting gpt-3 with "Let's think step by step" and then, after it thinks out loud, with "Therefore, the final answer is", increases the performance of gpt-3 very significantly. This seems to indicate that the algorithms we currently use benefit from using the [i/o space as cognitive workspace](https://twitter.com/karpathy/status/1529288843207184384). How much progress is likely to be made by giving large neural networks cognitive workspace that is more natural? Are there large language models today that have  internal state, and the ability reason on it for as long as they deem necessary (or at least to vary the amount of resources devoted) in order to come up with a better response?
# Utility functions and probabilities are entangled

*Originally posted as an* [*EA forum comment*](https://forum.effectivealtruism.org/posts/9pJ9CwsNmJQ8BmgSt/the-ea-movement-s-values-are-drifting-you-re-allowed-to-stay?commentId=kuh2ZSoDxmkh3eejE)*.*

Suppose that there are two effective altruist billionaires, April and Autumn. Originally they were funding AMF because they thought funding AI alignment would be 0.001% likely to work and solving alignment would be as good as saving 10 billion lives, which is an expected value of 100,000 lives, lower than they could get by funding AMF.

After being in the EA community a while, they switched to funding alignment research for different reasons.

*   April updated upwards on tractability. She thinks research on AI alignment is 10% likely to work, and solving alignment is as good as saving 10 billion lives.
*   Autumn now buys longtermist moral arguments. She thinks research on AI alignment is 0.001% likely to work, and solving alignment is as good as saving 100 trillion lives.

Both of them assign the same expected utility to alignment -- 1 billion lives. As such, they will make the same funding decisions. So even though April made an epistemic update and Autumn a moral update, we cannot distinguish between them from behavior alone.

This extends to a general principle: **actions are driven by a combination of your values and subjective probabilities, and any given action is consistent with many different combinations of utility function and probability distribution.**

As a second example, suppose Bart is an investor who makes risk-averse decisions (say, invests in bonds rather than stocks). He might do this for two reasons:

1.  He would get a lot of disutility from losing money (maybe it's his retirement fund).
2.  He irrationally believes the probability of losing money is higher than it actually is (maybe he is biased because he grew up during a financial crash).

These different combinations of probability and utility inform the *same* risk-averse behavior. In fact, probability and utility are so interchangeable that professional traders -- just about the most calibrated, rational people with regard to probability of losing money, and who are only risk-averse for reason (1) -- often model financial products as if losing money is *more likely than it actually is*, because it makes the math easier.[^gsh7xv180c8]

Implication
-----------

When observing that someone's behavior has changed, it's not obvious what changes are value drift vs. epistemic updates. You have to have some information besides behavior alone, or you'll have this extra degree of freedom in making interpretations.

[^gsh7xv180c8]: Formally, this is using the risk-neutral probability measure \(\mathbb Q\) rather than the real probability measure \(\mathbb P\) to price products; in finance circles it's apparently said that such people are living in "\(\mathbb Q\) world".
# NeurIPS ML Safety Workshop 2022

We're excited to announce the NeurIPS [ML Safety workshop](https://neurips2022.mlsafety.org/)! To our knowledge it is the first workshop at a top ML conference to emphasize and explicitly discuss x-risks.

X-Risk Analysis Prizes
----------------------

$100K in paper prizes will be awarded. There is $50K for best paper awards. There is also $50K in awards for discussing x-risk. This will be awarded to researchers who adequately explain how their work relates to AI x-risk. Analyses must engage with existing arguments for existential risks or strategies to reduce them.

What is the topic of the workshop?
----------------------------------

Broadly, the focus of the workshop is on [ML Safety](https://www.mlsafety.org/), which is the umbrella term that refers to research in the following areas:

**Robustness**: designing systems to be resistant to adversaries.

**Monitoring**: detecting undesirable behavior and discovering unexpected model functionality.

This category contains interpretability and transparency research, which could be useful for understanding the goals/thought processes of advanced AI systems. It also includes anomaly detection, which has been useful for detecting proxy gaming. It also includes Trojans research, which involves identifying whether a deep neural network will suddenly change behavior if certain unknown conditions are met.

**Alignment: **building models that represent and safely optimize hard-to-specify human values.

This also includes preventing agents from pursuing unintended instrumental subgoals and designing them to be corrigible.

**Systemic Safety: **using ML to address broader governance risks related to how ML systems are handled or deployed. Examples include ML for cyberdefense, ML for improving epistemics, and [cooperative AI](https://www.cooperativeai.com/foundation).

How do academic workshops work?
-------------------------------

The majority of AI research is published at conferences. These conferences support independently run workshops for research sub-areas. Researchers submit papers to workshops, and if their work is accepted, they are given the opportunity to present it to other participants. For background on the ML research community and its dynamics, see [A Bird's Eye View of the ML Field](https://www.alignmentforum.org/s/FaEBwhhe3otzYKGQt/p/AtfQFj8umeyBBkkxa).

Background
----------

A broad overview of these research areas is in [Unsolved Problems in ML Safety](https://arxiv.org/pdf/2109.13916.pdf).

For a discussion of how these problems impact x-risk, please see [Open Problems in AI X-Risk](https://www.alignmentforum.org/s/FaEBwhhe3otzYKGQt/p/5HtDzRAk7ePWsiL2L).
# «Boundaries» Sequence (Index Post)

In an upcoming short sequence of posts, I'm going to try to describe a causal pathway from 

1.  a key missing idea in the utility-theoretic foundations of game theory, leading to
2.  some problems I think I see in effective altruism discourse, leading further to
3.  gaps in some approaches to AI alignment, and finally, 
4.  implications for existential risk.

By default, I'll write one post for each of the above points, since they have different epistemic statuses and can be debated separately.  Posts 1 and 3 will be somewhat technical and research-oriented, and cross-posted to the alignment forum, whereas 2 and 4 will be non-technical and community-oriented, and cross-posted to the EA forum.  After that there might be more posts in the sequence, depending on the ensuing conversation.  In any case I'll try to keep this index post updated with the full sequences.

Here goes!

ETA Oct 31, 2022: There is now a proper sequence index for this sequence :)  
  
→ [«Boundaries» Sequence](https://www.lesswrong.com/s/LWJsgNYE8wzv49yEc)
# Norfolk Rationalists Weekly Wednesday Meetup @ Fair Grounds

Hello!

Come on out to Fair Grounds for good conversation, good food & drink, and a lovely atmosphere :)

When: We meet each Wednesday from 17:00-19:30 EDT.

Where: Fair Grounds, an excellent coffee shop located at 806 Baldwin Ave # 2, Norfolk, VA 23517. Do not park in the lot out front, you may get towed. Street parking is plentiful on the side streets off of Colley.

Who: Anyone who wants! Please feel free to come "even if you feel awkward about it, even if you’re not “the typical LW/EA/ACX reader”, even if you’re worried people won’t like you, etc."

Look for the ~two or more people actively talking and say hi. I'll try to grab the big round table. For the round table is the best table :)

Looking forward to hanging out and having most excellent conversations!

Cheers,  
Willa
# Seeking beta readers who are ignorant of biology but knowledgeable about AI safety

*Seeking the biology-ignorant! Come and do your part!*  
  
I am working on a post with Beth Barnes that will explain meiotic drive to AI safety people so they can determine if there are any useful analogies to gradient hacking. If you don't know what "meiotic drive" is, you might be perfect! I need beta readers who can tell me if they don't understand the biology stuff, so that I can get an idea of how they interpret the things they think they do understand, and so I know if what I'm saying is really making sense to them and seems related to AI safety concepts. I want this paper to be maximally accessible to AI safety researchers, so I want to really be sure I'm minimizing my biology jargon and explaining the necessary biology concepts well in language they understand.  
  
This would involve looking over a ~5-15 page google doc, flagging jargon you don't know or understand, and commenting on your confusions. Ideally I'd also find people who were willing to look at successive drafts. Ideally ideally I'd find people who were willing to have a brief call to talk about their interpretation of what they had read. The timeframe is within the next month (until end of August 2022).   
  
Comment below or PM me if you are interested :)
# For Better Commenting, Stop Out Loud

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6106508cccc3890b2479cc74a10093831abb30cea96cdcf1.png)

"Man stands face to face with the irrational. He feels within him his longing for happiness and for reason. The absurd is born of this confrontation between the human need and the unreasonable silence of the world." - *Albert Camus, The Myth of Sisyphus and Other Essays*

These days, a little unreasonable silence of the world actually sounds kind of nice! With apologies to Camus, I think a 21st-century Sisyphus would have been condemned to endlessly respond to annoying internet comments.

I'm not talking about outright trolling or doxxing. That's [21st-century Prometheus](https://astralcodexten.substack.com/).

Sisyphus isn't tormented by trollish insults, but by the compulsion to engage in ambiguous forms of effortful and frustrating dialog that he *knows* won't lead anywhere but feels like he has to participate in anyway. He's responding to comments full of ill-informed overconfidence, self-righteous activist monoperspective, and the crappiest of drive-by criticisms, which I call [PONDS](https://www.lesswrong.com/posts/k5TTsuHovbeTWgszD/for-better-commenting-avoid-ponds) ("prickly, opaque, nitpicky, disengaged, and shallow"). He himself, at some point in his life, has written a comment that's just as bad as the stupidity he's responding to, and he angrily empathizes with the other person's state of mind while wishing they'd just *listen and learn!* Every time he perfects his reply and hits "submit," the conversation rolls back downhill again.

Stopping Out Loud is a way of wedging the boulder of a bad conversation in place. Sisyphus declares that he will no longer be reading or replying to responses to the comment he's about to post. He's leaving the boulder halfway up the hill. Anyone trying to drag him down into a pointless fight or fruitless back-and-forth is, well, S.O.L.

Stopping Out Loud helps Sisyphus avoid the perception of social defeat that comes with letting the other person "have the last word." It also avoids disincentivizes further attention-getting and nastiness, and helps him feel like he's in control of his actions.

Sisyphus sometimes includes extra information when he Stops Out Loud.

*   Why he's stopping.
*   A clear, final statement of the point he wanted to make.
*   Conditions under which he'd revisit the thread.
*   An invitation or disinvitation to get in touch when he and the other person have had a chance to cool off.
*   A suggestion to continue the debate via PM, where it's sometimes easier to speak freely without feeling like the whole internet is looking on.

This is also helpful to the other person, who might just be engaging in [Socratic Grilling](https://slatestarcodex.com/2020/03/06/socratic-grilling/). It allows them to avoid wasting further effort in crafting a reply, and informs an immature but good-faith debater that they should consider approaching conversation differently next time. The audience also gains information about the way the conversation had on Sisyphus.
# AGI-level reasoner will appear sooner than an agent; what the humanity will do with this reasoner is critical

The recent advancements in language models, [Gato](https://www.deepmind.com/publications/a-generalist-agent), [Minerva](https://ai.googleblog.com/2022/06/minerva-solving-quantitative-reasoning.html), etc. suggest to me that there will be AGI-level "predictor"/"reasoner"/[generative model](https://www.lesswrong.com/posts/JqnkeqaPseTgxLgEL/conditioning-generative-models-for-alignment) sooner than an "agent". There is some subtlety though w.r.t. Gato, which is an "agent", but decidedly not an RL agent, it doesn't intrinsically value exploration, and it doesn't learn itself/bootstrap itself, and instead does imitation learning from other (actual RL) agents.  
  
I feel that this reasoner may suddenly generalise to astonishing capability, e. g. prove math theorems far beyond human comprehension, derive the theory of everything from data, and advance the state of the art in political science, ethics, and engineering, while remaining a non-agent and thus remaining relatively safe.  
  
I feel that it will be very important then to resist turning such an AI into an agent by attaching actuators to it and giving it tasks. However, this will be hard, because economic incentives for doing so will be enormous.

This shortform is probably a vague expression of the idea, which is much better thought through here: [Conditioning Generative Models for Alignment](https://www.lesswrong.com/posts/JqnkeqaPseTgxLgEL/conditioning-generative-models-for-alignment).
# Wanted: Notation for credal resilience

_Meta: I've spent 30-60 minutes thinking about this, and asking people who I'd expect to know about existing notation. I don't have scientific training, and I'm not active in the forecasting community._

# Problem

I want a clear shorthand notation for communicating [credal resilience](https://forum.effectivealtruism.org/topics/credal-resilience).

I want to be able to quickly communicate something like:


> My 80% confidence interval is 5-20. I think there's 10% chance I'd change my upper or lower bound by more than 50% of the current value if I spent another ~day investigating this.

I’m using the term “credal resilience”. Some people call this "robustness of belief".


# Existing notation for confidence intervals

[APA style guide](https://blog.apastyle.org/apastyle/2010/06/formatting-statistics-using-brackets.html) suggests the following:

> 80% CI [5, 20]

This seems like the best and probably most popular option, so let's build on that.


# Proposal

For clarity, I'll repeat the example I gave above:

> My 80% confidence interval is 5-20. I think there's 10% chance I'd change my upper or lower bound by more than 50% of the current value if I spent another ~day investigating this.

To communicate this, I propose:

> 80% CI [5, 20] **CR [0.1, 0.5, 1 day]**

Low numbers in the first two parameters indicate high credal resilience. The unit of additional investigation tells you the approximate cost of further investigation to "buy" this extra information.

You can specify hour / day / week / month / year for the unit of additional investigation.

# Thoughts?

I'd love to hear people's thoughts on this. Two questions I'll highlight in particular:

1. Do you think it'd be worth developing a good notation for credal resilience, then popularising it?

2. What do you think of my particular proposal? What might be better?
# Perverse Independence Incentives

In writing about parenting, I don't write enough about what didn't work: this might give the impression that everything I try works, which is not the case! Here's an example of something that didn't, and which I might have caught before trying if I'd thought a bit more.

Brief recap: I see one of my main roles as a parent being [building independence](https://www.jefftk.com/p/growing-independence):

> I want to let my kids be as independent as possible, as early as possible. Not only does it make their lives better, because they can meet their own needs how they want, but it makes my life easier, because they can handle more on their own. Sometimes this involves a bit more effort up front, but I think it's substantially less effort in total.

For a while I had an approach which was essentially, "if you can do it yourself I'm not going to do it for you." For example, soon after Lily learned how to get her own cereal, we had some conversations like:

> Lily: Papa, can I have some cereal?
> 
> Me: You know how to do that!
> 
> Lily: I don't want to, and you used to get it for me.
> 
> Me: Now that you can do it I'm not going to do it for you.
> 
> Lily: \[unhappy\]

After a few weeks I realized this was counterproductive. Yes, it did mean more practice at new skills, and it is where I want us to end up eventually, but the incentives are all wrong. Instead of her former excitement at learning new things and expanding what she was capable of, each new skill now made her life harder because I was less helpful. You can sort of think of this as too high a marginal tax rate: I was claiming too much of the surplus from her investments in her abilities.

After I realized it wasn't working I toned this down a lot. I will still sometimes refuse to do things they know how to do themselves, but most of the time they're happy enough to do the thing on their own, whenever they want, at their own speed, and in they way they prefer, that I don't need to push them along.

One place where I'll still consistently refuse is when I wouldn't have done it for them even before they learned how. For example, if they don't like dinner and ask me to heat up something else I'll do that, but then if they don't like that I won't just keep getting them additional things. Now that she can use the microwave, however, Lily has an additional option, which is that she can go heat herself something up on her own.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0dRsqvXjyJbSeiUWV9FZAxL2TyWf31Zozf7yoahBw85winWiSEEgcJT8xAjJULpZpl)*
# Astral Codex Ten Berlin Meetup


This meetup is an open get-together for readers of Scott Alexander's blog [Astral Codex Ten](https://astralcodexten.substack.com/). We meet in Tiergarten and sit either on comfy grass or on blankets that you bring with you. If you want to make sure you have a blanket, take agency and bring your own. The same goes for everything else you expect to want to have access to whether food, drinks, mosquito repellents spray, temperature-appropriate clothing, sunglasses, a sunhat, or more creative items. Be agenty ;)

You can find the exact location in Tiergarten either through [meetup.com](https://www.meetup.com/lesswrong-rationality-waitbutwhy-slatestarcodex-berlin/events/287542935/) or at [w3w.co/alert.pencils.doctors](https://w3w.co/alert.pencils.doctors)
# Is there any writing about prompt engineering for humans?

In much the same way that we need to prompt engineer language models to get them to handle our questions correctly, we know that humans respond differently to "This intervention will save 200 lives" and "This intervention will result in 400 deaths" when talking about a population of 600 people. 

Is there any pre-existing writing that touches on this?
# One-day applied rationality workshop with Duncan Sabien

I invited Duncan Sabien to give a workshop the day after LWCW (and I'm doing the organizational background work).

If you want to read more and sign up, the description is at https://www.lesswrong.com/posts/pEFzqeTCBfo4nrN47/one-day-applied-rationality-workshop-in-berlin-aug-29-after
# I want to donate some money (not much, just what I can afford) to AGI Alignment research, to whatever organization has the best chance of making sure that AGI goes well and doesn't kill us all. What are my best options, where can I make the most difference per dollar?

I don't understand this field well, I'm hoping you guys can help me out.
# What are the Red Flags for Neural Network Suffering? - Seeds of Science call for reviewers

[Seeds of Science](https://www.theseedsofscience.org/) is a new journal (funded through Scott Alexander's [ACX grants program](https://astralcodexten.substack.com/p/acx-grants-results)) that publishes speculative or non-traditional articles on scientific topics. Peer review is conducted through community-based voting and commenting by a diverse network of reviewers (or "gardeners" as we call them). 

We just sent out an article for review - "What are the Red Flags for Neural Network Suffering?" - that may be of interest to some in the LessWrong community, so I wanted to see if anyone would be interested in joining us a gardener to review the article. It is free to join and anyone is welcome (we currently have gardeners from all levels of academia and outside of it). Participation is entirely voluntary - we send you submitted articles and you can choose to vote/comment or abstain without notification (so it's no worries if you don't plan on reviewing very often but just want to take a look here and there at what kinds of articles people are submitting). Another unique feature of the journal is that comments are published along with the article after the main text. 

To register, you can fill out this [google form](https://docs.google.com/forms/d/e/1FAIpQLSfRIicHT7jIZcSUjwsIlby6JBxx2ZVeD5kseZBpgGFtp8pLfg/viewform). From there, it's pretty self-explanatory - I will add you to the mailing list and send you an email that includes the manuscript, our publication criteria, and a simple review form for recording votes/comments.

Happy to answer any questions about the journal through email or in the comments below. Here is the abstract for the article. 

**What are the Red Flags for Neural Suffering?**
------------------------------------------------

By \[redacted\] and \[redacted\]

**Abstract:**

Which kind of evidence would we need to see to believe that artificial neural networks can suffer? We review neuroscience literature, investigate behavioral arguments and propose high-level considerations that could shift our beliefs. Of these three approaches, we believe that high-level considerations, i.e. understanding under which circumstances suffering arises as an optimal training strategy, is the most promising. Our main finding, however, is that the understanding of artificial suffering is very limited and should likely get more attention.
# Clapping Lower

Clapping is normally relatively high pitch:

([mp3](https://www.jefftk.com/three-high-claps.mp3))  
[![](https://www.jefftk.com/high-clap-spectrum.png)](https://www.jefftk.com/high-clap-spectrum-big.png)

I find clapping annoyingly high, so I usually clap a bit differently:

([mp3](https://www.jefftk.com/three-low-claps.mp3))  
[![](https://www.jefftk.com/low-clap-spectrum.png)](https://www.jefftk.com/low-clap-spectrum-big.png)

If you also want to clap lower, the main idea is to maximize the volume of air you trap between your clapping hands. Arch the palms, and try to have the full circumference of your palms collide. It takes some practice, but after a bit it should come pretty naturally.

This is also useful musically, if you're clapping along and want to clap low for some beats (ex: 1 and 3) and high for others (ex: 2 and 4).
# Two Kids Crosswise

When traveling, we sometimes end up somewhere with the older two kids sharing a double bed. Initially we would put them down the normal way, as the bed was made to be used, but recently we realized it goes much better if you think of the bed as being sideways:

[![](https://www.jefftk.com/sleep-kids-sideways.png)](https://www.jefftk.com/sleep-kids-sideways-big.png)

Kids are pretty short, and if they sleep crosswise they're farther from each other, which minimizes fighting. Our kids don't fight that much normally, though they don't get along as well as they did a year ago, but bedtime while traveling is one of the worst situations: they're extra sleepy, and the sleepier they are both the worse their impulse control is and the more irritable they are. Rotating the arrangement makes it easy for them each to have their own space. Separate sheets and blankets also help.

A [Full](https://en.wikipedia.org/wiki/Bed_size#North_America) is 54" wide and a Queen is 60" wide, so the [median](https://www.cdc.gov/growthcharts/data/set1clinical/cj41l021.pdf) [child](https://www.cdc.gov/growthcharts/data/set1clinical/cj41l022.pdf) gets too tall for this at about 9y on a Full and 12y on a Queen.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid034HfrrLh3F7BpgWj13gw9JxErW6ooVSKgmfS5qfRbc8pkerhuMv2BiZGxn4Bdo4Cbl)*
# PredictIt is closing due to CFTC changing its mind

[https://www.predictit.org/platform-announcements](https://www.predictit.org/platform-announcements)

> The staff of the Commodity Futures Trading Commission (CFTC) has [withdrawn](https://www.cftc.gov/PressRoom/PressReleases/8567-22) the No-Action letter (NAL) issued to Victoria University of Wellington and under which PredictIt has operated since 2014.

What does the LW community think about this happening for PredictIt in particular, and prediction markets in general?
# Careful with Caching

A friend recently shared this graphic on Facebook:

[![](https://www.jefftk.com/prison-vs-college-wrong.jpg)](https://www.jefftk.com/prison-vs-college-wrong-big.jpg)

(I've added the "wrong" overlay.)

This is clearly incorrect: there's no way Massachusetts has more prisons than colleges. (MA actually has the largest ratio of colleges to prisons in the US.) After putting a link to the [original source](https://studee.com/discover/usa-prison-v-college/) in the Facebook discussion, however, we found something pretty weird: people on mobile were seeing the incorrect map, but people on desktop were seeing a corrected one:

[![](https://www.jefftk.com/prison-vs-college-right.jpg)](https://www.jefftk.com/prison-vs-college-right-big.jpg)

It turns out that Facebook was appending a tracking parameter, `?fbclid=...` on desktop, but not on mobile. Normally this wouldn't do anything, because the site would ignore that in determining what page to return, but this site is apparently configured with a cache.

Many sites use caches to make it easier to serve pages. If you ask them for a page they'll give it to you (which might require a lot of work to generate) and then save a copy. Then when someone else asks for the same page, they can return the saved copy instead of putting in all that work to regenerate it. The site has a cached copy of `/usa-prison-v-college` in its cache with outdated content, but since `?fbclid=...` is always followed by a new token those requests will never be found in the cache, and they return the current, corrected, page.

If you have a cache, what do you do when you change the page, like they did here? There are two main approaches:

*   Have a way to tell the cache the page has changed and it should forget its copy.
    
*   Always cache for a short time. Even just one minute can take a lot of load off a server that is getting thousands of requests for a hot page.
    

In this case they apparently didn't do either: we were running into this yesterday, and as of right now the site is still returning incorrect data.

(On the original question, comparing the number of colleges to the number of prisons is pretty silly: if one state runs large prisons and small colleges is that any better than a state that does the reverse? Comparing the number of people in prison vs college would make much more sense.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02638UgnJEVMKuEQfeJ1sNgRB9kxQRKCe76ZYpCFJi3VCL5fSNpvicA2VCiGEH67Kil)*
# Can we get full audio for Eliezer's conversation with Sam Harris?

I listened to [this full interview](https://www.samharris.org/podcasts/making-sense-episodes/116-ai-racing-toward-brink) in 2018, and it made me take AI risk way more seriously. I now plan to devote my career to this problem, so people often ask me what the concern with AI is all about. *I really really really want to share this podcast with them, but these days less than half of the audio is available for free* (the full interview requires a [subscription of at least $8.33 per month](https://www.samharris.org/subscribe)). So the best I can do is offer them [this transcript](https://intelligence.org/2018/02/28/sam-harris-and-eliezer-yudkowsky/), which I find much less engaging than the audio. Sam appreciates effective altruism and has [no problem releasing full audio for episodes about gun violence](https://www.youtube.com/watch?v=f7hyxU-zoLk), so I bet he'd have no problem releasing the full Eliezer Yudkowsky audio if someone (e.g. Rob Bensinger) prodded him. To do my part, I just sent a request via [Sam's contact form](https://www.samharris.org/contact).  
  
Side note: if Eliezer did some new interviews on relevant podcasts, he might persuade a lot more people to take alignment seriously. And there would be a more up-to-date interview (citing more attention-grabbing evidence for the possibility of advanced AI, e.g. DALL·E 2) for me to share with my vaguely-interested-but-mostly-unconcerned friends.
# Experiment: Be my math tutor? 

I'm a CS student at a top university who takes AI x-risk incredibly seriously. While university has given me a lot of fun opportunities to grow, I am not entirely satisfied with the rate at which my skill is improving.

If I want to stop human extinction, I'll need to \[go faster\](https://sive.rs/kimo). This is where you can help.

Details:

*   Subjects: problem solving skills, algebra, combinatorics, information theory, probability & statistics, category theory. Subjects are flexible. 
*   Hours are about 5 hours / week. Scheduling is highly flexible.
*   Able to meet over Zoom, Google Meet, or Signal.
*   Rate is between $40 and $100 per hour, negotiable and depending on fit and experience.

Please comment or DM if you're interested! Feel free to just send me a copy of your resumé or CV, even if it's not completely up to date.

Finally, if you're on the fence about reaching out, please do. Even if we don't end up being a good fit, I'll still be happy to meet you. 

Edit: Decreased upper range of rate from $130 to $100, after realizing how weak my home currency is against the dollar. Rate is still negotiable depending on fit and experience.
# ACX Montreal Meetup August 21 2022 **Venue Changed**

**Venue: **

Due to projected rain, **the venue has been changed to to the nearby** [**Olde Orchard pub & grill**](https://www.google.com/maps/place/Ye+Olde+Orchard+Pub+%26+Grill/@45.5134983,-73.5727146,17z/data=!3m1!4b1!4m5!3m4!1s0x4cc91a35812fae03:0xf5fdb66587011cc4!8m2!3d45.5135005!4d-73.5727198) **at 20 Prince Arthur street W.**  
Date & Time: August 21st, 2022 2pm.

**Description:**

Come on out to the ACX (Astral Codex Ten) Montreal Meetup. We will be hosting a discussion about an intriguing and exciting topic: the Lindy effect.

Simply put, the Lindy effect states that the longer something has existed, the longer it will continue to exist. The lifespan of non-biological entities increases as time goes on, instead of decreasing. If we keep this in mind, we might ignore the newest trends and decide to spend our time on things that have withstood the test of time.

Is this a valuable way to make decisions? Is the Lindy effect compatible with a rationalist worldview or mere pseudoscience? We have selected two articles: one to explain the Lindy effect and one to provide an example of how it can be applied. Doing the reading is recommended but feel free to come to the meetup even if you forget or don't have time to.

[Lindy effect](https://en.wikipedia.org/wiki/Lindy_effect) on Wikipedia (4m read)  
[Ceiling Height Matters](https://web.archive.org/web/20210218190030/https://paulskallas.substack.com/p/ceiling-height-matters) by Paul Skallas (10m read)

RSVPing is not mandatory but it helps with planning the event and encourages others to come.
# What are some Works that might be useful but are difficult, so forgotten?


  
It seems like the internet in general, and social media broadly, e.g. LessWrong, has this problem that there's a lot of data. The data has to be filtered somehow. The main way data is filtered is by (1) recency, (2) backlinks, and (3) upvotes.

One problem is that these filters don't distinguish between stuff that's not upvoted or backlinked because it's not so good / interesting / important / novel / true / useful / etc., vs. stuff that's not upvoted or backlinked because it was hard to understand (or, easy to misunderstand as not good) and therefore didn't meet the threshold for being signal boosted. In other words, we aggregate signals of "I don't know about this yet" roughly the same way we aggregate signals of "This isn't good enough that it should be remembered much".

This seems like potentially a big problem, or rather, a big opportunity. Maybe there's good ideas that didn't spread much that we could revive. Maybe ideas would develop faster if people could expect that other people would remember "there's an open question here" rather than just not remembering. But IDK how big of an opportunity it is, so I wonder:

What are some examples of works (of any kind whatsoever) that were / are hard to understand, and so didn't get signal boosted when they were new enough to have attention directed to them to become understood, but instead got lost in the shuffle?

One of the famous examples around here is \[Infrabayesianism\](https://www.alignmentforum.org/posts/83DimRqppcaoyYAsy/job-offering-help-communicate-infrabayesianism). Maybe some of \[Donald Hobson's work\](https://www.lesswrong.com/users/donald-hobson) gives examples. What else?
# What are some alternatives to Shapley values which drop additivity?

I originally asked this on math.stackexchange; after reading Diffractor's Unifying Bargaining sequence ([Part 1 here](https://www.lesswrong.com/posts/rYDas2DDGGDRc8gGB/unifying-bargaining-notions-1-2)) I'm wondering if there are more insights floating about, so I'm repeating it here.

Shapley values seem to be the standard answer to "how should a coalition split the rewards of their cooperation", but I'm curious about alternatives.

The [standard characterization of Shapley values](https://en.wikipedia.org/wiki/Shapley_value#Characterization) says that Shapley values are the unique coalition payments which satisfy a bunch of properties. Three of them (efficiency, symmetry, and null player) seem pretty necessary for any "reasonable" or "practical" coalition payment rule, but the last one (linearity) does not.

If I didn't care for linearity (or its close synonyms, additivity and aggregation):

*   What sorts of payment rules become available?
*   What other properties of Shapley values are maintained?
*   What other properties would produce a uniquely characterized payment rule?

Alternatively, are any of the other properties also reasonable to drop (for instance, symmetry)? What do you end up with?
# Progress links and tweets, 2022-08-09

**Links**
---------

*   [The challenge of corrosion and how engineers fight it](https://youtu.be/2RbiCOFffRs) (YouTube)
*   [Arnold Kling claims that we *over*-consume medical services in the US](https://arnoldkling.substack.com/p/crisis-of-abundance-81)
*   [The lessons of Xanadu](https://jasoncrawford.org/the-lessons-of-xanadu) (on my personal blog)

**Queries**
-----------

*   [Has anybody done serious, detailed work on the mechanisms that drive technological s-curves?](https://twitter.com/Ben_Reinhardt/status/1556290357897338880) (@Ben_Reinhardt)
*   [Who are the smartest people working on battery storage? Best writing on the topic?](https://twitter.com/juliadewahl/status/1554199811980533760) (@juliadewahl)
*   [What are examples of projects getting done really fast by organizations that are normally very slow?](https://twitter.com/_brianpotter/status/1554168494157168641) (@_brianpotter)
*   [Is there a Thiel Fellowship type program for PhD candidates to bail out of academia?](https://twitter.com/tayroga/status/1553868627450814464) (@tayroga)

**Tweets & retweets**
---------------------

*   [“Appeal to nature” fallacies are ethically problematic for progress](https://twitter.com/RafaRuizdeLira/status/1554556129597722625) (thread by @RafaRuizdeLira)
*   [Josh Barro on Matt Yglesias on “induced demand”](https://twitter.com/jbarro/status/1554819854603001857) (@jbarro)
*   [US immigration as an important-but-not-urgent policy disaster](https://twitter.com/sama/status/1554459751353683969) (@sama)
*   [The US basically stopped building large-scale water infrastructure in the 1980s](https://twitter.com/_brianpotter/status/1554521379319517186) (@_brianpotter)

[![](https://pbs.twimg.com/media/FZLE19oXoAc0MoX.png)](https://pbs.twimg.com/media/FZLE19oXoAc0MoX.png)
# Emergent Abilities of Large Language Models [Linkpost]

I've [argued before](https://forum.effectivealtruism.org/posts/7ZZpWPq5iqkLMmt25/aogara-s-shortform?commentId=NKwpLDfJ4PYZkmtxs) against the view that intelligence is a single coherent concept, and that AI will someday suddenly cross the threshold of general intelligence resulting in a hard takeoff. This paper doesn't resolve that debate entirely, but it provides strong evidence that language models often have surprising jumps in capabilities. 

From the abstract: 

> Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence implies that additional scaling could further expand the range of capabilities of language models.

Key Figures:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/260dfa9c768cac039fcd8d4d43e4e52726d15b37c456adea.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/046ce54f5bd2c1bfc1c79add1ff4fdf26a22c9b1bb5a8362.png)

Related: [More is Different for AI](https://www.lesswrong.com/posts/Lp4Q9kSGsJHLfoHX3/more-is-different-for-ai), [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://arxiv.org/pdf/2201.02177.pdf), [Yudkowsky and Christiano on Takeoff Speeds](https://www.lesswrong.com/posts/vwLxd6hhFvPbvKmBH/yudkowsky-and-christiano-discuss-takeoff-speeds#_Yudkowsky__17_22_)