Harley-ml/lesswrong huggingface dataset:
# A compressed take on recent disagreements

An underlying generator of many recent disagreements appears to be differing locations of the disagreeing individuals on the [Yudkowsky-Hanson spectrum](http://intelligence.org/files/AIFoomDebate.pdf). That is, differences in individuals' distributions over the background variable "how does an optimizer's ability to optimize scale as you apply (meta) optimization to it". I'm going to simplify my discussion by boiling this variable down to an "expected steepness" (as opposed to a distribution over optimization-in vs. optimization-out curves).

Eliezer believes this relationship to be steeper than do many others in the alignment sphere. For instance, many (~half by my count) of the disagreements listed in [Paul's recent post](https://www.lesswrong.com/posts/CoZhXrhpQxpy9xw9y/where-i-agree-and-disagree-with-eliezer) appear to pretty directly imply that Paul believes this relationship is much less steep than does Eliezer. It seems likely to me that this difference is the primary generator of those disagreements.

Eliezer has previously suggested formalizing this "optimization-in vs. optimization-out" relationship in [Intelligence Explosion Microeconomics](https://intelligence.org/files/IEM.pdf). Clearly this is not such an easy thing, or he probably would have just done it himself. Nonetheless this may be a pathway towards resolving some of these disagreements.

So, how *do* optimizers scale with applied optimization?
--------------------------------------------------------

I'll quickly give my two cents on the matter by noting that we live in a world where the best mathematicians are, quite literally, something like 1000 times as productive as the average mathematicians (keeping in mind this gap only spans about a quarter of the range of variation of mathematical ability in the modern human population; about +3 to +6 stdevs).

This huge difference in observed performance occurs despite the minds in question having about the same amount of hardware (brain size) built out of components running at about the same speed (neurons) executing the same basic algorithm (genetic differences are tiny compared to size of genome). The difference appears to be largely a result of small algorithmic tweaks.

This leads me to the conclusion that, if you take a ~human level AGI and optimize it a little bit further, you probably end up with something pretty strongly superhuman. It also leads me to conclude that it's hard to build a ~human level AGI in the first place because it's a small target on the capability axis: you'll likely just accidentally blow past it, and even if you can hit it, someone else will just blow past it soon thereafter.
# Norfolk Rationalists Weekly Wednesday Meetup @ Fair Grounds

Hello!

Please come on out to Fair Grounds for good conversation, good food & drink, and a lovely atmosphere :)

When: We meet each Wednesday from 17:00-19:30 EDT.

Where: Fair Grounds, an excellent coffee shop located at 806 Baldwin Ave # 2, Norfolk, VA 23517. Do not park in the lot out front, you may get towed. Street parking is plentiful on the side streets off of Colley.

Who: Anyone who wants! Please feel free to come "even if you feel awkward about it, even if you’re not “the typical LW/EA/ACX reader”, even if you’re worried people won’t like you, etc."

Look for the ~two or more people actively talking and say hi. I'll try to grab the big round table.

Looking forward to hanging out and having most excellent conversations!

Cheers,  
Willa
# Predicting Parental Emotional Changes?

Before I became a parent I cared about children in general, intellectually, in the same way that I care about people in general: a general sense that I want good things to happen to them. Over the first few days of Lily's life, however, I noticed I cared for her deeply and emotionally, where her wellbeing was incredibly important to me, and she was far more cuddly and interesting than I expected. Over the next few months, I realized that my disposition toward children in general had also changed, and while I didn't feel as strongly as toward my own, they were now similarly far more fun, interesting, and precious than they had been. I also started having [strong emotional reactions](https://www.jefftk.com/p/parenting-and-emotional-changes) to reading about harm to children. Paul Graham [writes](http://www.paulgraham.com/kids.html) about a similar experience, though I think one a bit more sudden than my own:

> Partly, and I won't deny it, this is because of serious chemical changes that happened almost instantly when our first child was born. It was like someone flipped a switch. I suddenly felt protective not just toward our child, but toward all children.

And Maia [commenting](https://juliawise.net/parenting-expectations-and-experiences/#comment-621) on a post of Julia's:

> For me it was like a switch flipped when I gave birth. The mental change felt similar to the mental change I went through during puberty: just adding a completely new biological drive I didn't have before. I actually didn't find babies very cute or interesting at all before, but I found my baby extremely so immediately

In deciding to have kids this is a very difficult factor to consider, because it doesn't happen to everyone. I have a vague impression that it is a bit more common in men than women, perhaps because women understandably have a higher threshold for deciding to have kids, but there are definitely men who expect it to happen and are dismayed when it doesn't and women who are surprised at how much their feelings toward children change.

Getting better at predicting this transformation seems valuable: since becoming a parent is a lifelong decision, the value of information is really high. Do people tend to have similar experiences to their own parents? Is it correlated with any other aspects of personality? Can your friends or other people who know you well predict this? Can it be triggered, if it's going to, by something less permanent than becoming a parent or is the permanence a key component? Is it something that may or may not happen, and when it happens it is a similar magnitude for everyone, or is it something where most people change a bit on becoming a parent, anywhere along a range from "imperceptibly" to "enormously"? Do we know how prevalent these changes are?
# LessWrong Meetup - Changing our Minds

One of the core marks of the rationalist is the ability to change one's mind. In the workshops, we will explore how we changed our minds in the past.  
We will share insights with each other and look at which ways of changing our minds work for us.

16:00-16:15 Arriving and chatting  
16:15-17:15 First Workshop Block  
17:15-17:45 Mingling  
17:45 -19:15 Second Workshop Block  
19:15 -20:00 Mingling

Ideally, you bring pen&paper but you can also use your phone to write things down.

Visit [Meetup](https://www.meetup.com/lesswrong-rationality-waitbutwhy-slatestarcodex-berlin/events/286898836/) for the exact location.
# What New Desktop Should I Buy?

It's time for me to buy a new desktop, as my old one isn't working so well anymore. 

It also happens to be 'black Friday in July' which opens up the opportunity to perhaps get a good sale price somewhere.

Time is valuable, so I'm happy to spend a reasonably large amount if it actually matters, but I don't want to throw good money after nothing.

Given how many people would benefit from getting this question right, and how many people are likely to have good answers, I figured I'd ask here.

My specific goals are basically:

1.  Windows 11, I'm not open to negotiation on this one for various reasons.
2.  Will be able to open and rapidly switch between a ton of chrome tabs, including massive google sheets and google docs.
3.  General reliability and future-proofing. 
4.  High-end gaming would be nice but mostly reliable medium-end gaming is fine.
5.  Handle at least 3+ monitors well.
6.  Relatively quiet fan is a big plus, especially loud ones are deal breakers.
7.  The usual other stuff you'd want a computer to be able to do these days.

Current top candidate after talking with one friend first is [this Alienware PC](https://www.dell.com/en-us/shop/gaming-and-games/alienware-aurora-r13-gaming-desktop/spd/alienware-aurora-r13-desktop/wdr13aur30h?view=configurations&configurationid=b767d88d-a71e-4495-9d8a-381c31712cf5#features_section) configured with processor 12th Gen Intel® Core™ i7-12700KF (25 MB cache, 12 cores, 20 threads, 3.60 to 5.00 GHz Turbo), Win11 Pro, NVIDIA® GeForce RTX™ 3080, 10 GB GDDR6X, LHR, 64 GB memory, HDD 1 TB, M.2, PCIe NVMe, SSD (with plan to also use the existing computer's HD as well) and Office Student.

Secondary possibility is Lenovo is having a massive sale on [that includes the P620](https://www.lenovo.com/us/en/p/workstations/thinkstation-p-series/thinkstation-p620/wmd00000429), seems like the more expensive versions are way more expensive than they need to be even post-sale so I'm suspicious (my old computer is a Lenovo and mostly was fine for a while until it wasn't). 

Also very open to additional brands/possibilities.  

UPDATE: Someone has volunteered to build it on my behalf, so yay. Still seems like a good question in general.
# Avoid the abbreviation "FLOPs" – use "FLOP" or "FLOP/s" instead

Especially in discussions about AI, the abbreviation "FLOPs" is being used for both "floating point operations per second" (a measure of computational power) and "floating point operations" (a measure of total computations, and equivalent to the previous term times seconds). This is ambiguous and confusing. For clarity, I propose people avoid this specific abbreviation and instead use the alternatives of "FLOP" (for floating point operations) and "FLOP/s" (for floating point operations per second).
# Checksum Sensor Alignment

# Problem

Suppose you have a powerful AGI protecting a diamond in a vault[^1]. You have various sensors in the room, all monitoring the diamond. There is a camera, watching the diamond. There is a pressure plate under the diamond confirming it is still in its stand. There are a total of 10 sensors. The powerful AGI can hack all 10 of your sensors.

How do you align the AGI such that it does not hack your sensors? You cannot tell (directly) whether the AGI believes the diamond is in the vault. You must describe your instructions to the AGI in terms of the sensor readings.

[^1]: This is a variant on [ARC's ELK competition](https://www.lesswrong.com/posts/QEYWkRoCn4fZxXQAY/prizes-for-elk-proposals).

# Solution

Instruct the AGI to perform the simplest series of actions that cause 9 of your sensors to agree and 1 of your sensors to disagree. Suppose (without loss of generality) that the sensor which disagrees is your camera. Either the AGI hacked the camera or the AGI has hacked your other nine sensors. The AGI has probably just hacked your camera. Carefully examine what the AGI did to hack your camera. Fix the vulnerabilities.

Repeat the above process until the AGI can no longer hack your sensors.
# How do AI timelines affect how you live your life? 

This question is more about personal decision-making rather than for example deciding to work on AI safety for altruistic reasons. If I were thinking about this from a purely selfish perspective, it seems pretty likely that if I expect transformative AI to arrive in 20 years, I should live my life a bit differently than people who don't expect to get TAI in their lifetimes.   
  
I'm curious about if your own beliefs on AI timelines have affected anything that you do in your personal life - perhaps decisions related to saving money, personal relationships, health etc.
# Rainmaking

There is an old TV show called 'The Wire', and in it a con artist fleeces a mark.

The mark's friend tries to wake him up, whereupon the mark insists that the con artist has helped him.  The mark's friend tells him to 'wake up, he rain-made you!'

"What?"

"A guy says if you pay him, he can make it rain.  You pay him.  If and when it rains, he takes the credit.  If and when it doesn't, he finds reasons for you to pay him more."

This is a neat tv show moment, to be sure, but it is also a pattern that you will find everywhere in the corporate world.

A given product has a few devs, who do the work, a few bosses, who fire devs that suck, and a squad of rainmakers who mess around on their phones.  They are called Security Consultants, Architects, Scrum Masters... but what they all have in common is the same thing that an old saying recites about political doctrines.

They can never fail, only be failed.  Paying security guys doesn't stop you from getting hacked.  Paying <insert political trend> trainers doesn't stop you from getting sued.  There is, in fact, no world state that can ever indicate that paying one of these folks was a mistake.  Either their flag is checked, in which case all is well, or it is not, in which case they need more resources.

Lesswrongers will be familiar with the old concept of 'make your beliefs pay rent'.  It is terrifying how rare basic comprehension of this notion is in the organizations we depend upon.  People will accept unfalsifiable axioms as doctrine without the slightest qualm, and then you are gritting your teeth through another round of <insert buzzword> exercises.

They trade upon the credit earned by the few legitimate practitioners of their occupations, who genuinely do consult on security or master scrums.  It can be hard to tell the two apart, to grok whether your rainmaker is toting a cloud seeder or thoughts and prayers.

The only thing I've ever seen work was essentially formalizing the above.  Have everyone give 'failure statements' of the form "I will have failed if at time X Y measurable statement is true"  and look at the nature of the statements.  Productive people's will be about their contribution, rainmakers will talk about the weather.
# How do you concisely communicate & navigate the politics / culture at your job working at a large corporation or institution?

Previously I've worked at small companies and state government or educational institutions. Now I work at a ~30,000 person private corporation and need to better adapt to that cultural situation.

My biggest cultural struggles so far are communicating concisely to people at multiple levels in the hierarchy across multiple different teams and knowing how to navigate the politics of a large corporation (I'm not used to directors being meaningfully different from managers, or the massive differences between what's called "upper management", "management", and everyone else).

I work in a technical role and support customers across every rank & role in the enterprise.

What are your tips, tricks, and advice for handling concise communication & navigating politics / the culture at a large corporation or institution?
# Progress links and tweets, 2022-07-12

**Links**
---------

*   [Patrick Collison on universal covid vaccines: we could have them very soon, if it weren’t for bureaucratic roadblocks](https://www.slowboring.com/p/we-could-have-universal-covid-vaccines)
*   [“Current funding organizations cluster heavily around a few management styles, warping the research ecosystem”](https://parpa.substack.com/p/bottling-lightning) (Ben Reinhardt)
*   [The Tony Blair Institute for Global Change announces their Progress Fellows Programme 2022](https://institute.global/policy/progress-fellows-programme-2022) (h/t [@benedictcooney](https://twitter.com/benedictcooney/status/1544709779012059140))
*   [“A jeremiad is a long literary work… in which the author bitterly laments the state of society and its morals in a serious tone of sustained invective, and always contains a prophecy of society’s imminent downfall.”](https://en.wikipedia.org/wiki/Jeremiad) (Sad and amusing to me that there is a word for this!)

**Queries**
-----------

*   [What would you change the h-index to?](https://twitter.com/eric_is_weird/status/1542216618817863681) (@eric\_is\_weird)
*   [Who should be in a “Most Interesting People in the World” speaker series?](https://twitter.com/tom_morganKCP/status/1545740970297368585) No one has suggested me yet… (@tom_morganKCP)

**Retweets**
------------

*   [World’s Fair posters from DALL-E](https://twitter.com/arbesman/status/1541434872421654530) (@arbesman)
*   [The thinking machines allow us to become more human](https://twitter.com/tszzl/status/1542230536789164032) (@tszzl)
*   [Why did we stop reclaiming land for cities?](https://twitter.com/MTabarrok/status/1541897992315322369) (@MTabarrok)
# Straw-Steelmanning

I've noticed that when people are asked to "Steelman" a position, they sometimes instead do what I would call "Straw-Steelmanning". Someone can also straw-steelman without having been asked to steelman or having said that they would do so. 

What is straw-steelmanning? Assume someone makes an argument X for a claim C, and you are arguing against X.

*   **Straw-manning (bad)**: You replace X with a weaker argument Y and argue against that, pretending as if you have thereby refuted X.
*   **Steel-manning (good)**: You replace X with a stronger argument Y which still contains the core of X and argue against that, thereby actually refuting X. (The term can also be used in a context where you are not actually arguing against the claim C).
*   **Straw-steelmanning (bad)**: You replace C with an entirely different claim D and make an argument Y for it which you consider to be stronger than X, pretending as if you no longer need to argue against C.

An example which I have noticed is something like the following:

*   "Can you steelman the position that future AI systems will pose an existential risk?"
*   "Well, while we whould not take these hollywood movie plots seriously, there are real social problems with AI that we have to deal with. AI will potentially cause massive inequality because entire industries will be automated and a small amount of corporation will own the AI tools that facillitate that. AI engineers will earn large wages, while demand for other professions stagnates. Moreover, we need to worry about biases in AI systems, because \[etc, proceeds to argue more\]"

This is a straw-steelman, because they have

*   Simply bypassed the original claim, replacing it with a different claim that they already agreed with.
*   Proceeded to argue for that claim, ignoring the original.
# John von Neumann on how to safely progress with technology

From a 1955 article written by von Neumann titled, "[Can We Survive Technology?](http://geosci.uchicago.edu/~kite/doc/von_Neumann_1955.pdf)"

> What safeguard remains? Apparently only day-to-day — or perhaps year-to-year — opportunistic measures, along sequence of small, correct decisions. And this is not surprising. After all, the crisis is due to the rapidity of progress, to the probable further acceleration thereof, and to the reaching of certain critical relationships. Specifically, the effects that we are now beginning to produce are of the same order of magnitude as that of "the great globe itself." Indeed, they affect the earth as an entity. Hence further acceleration can no longer be absorbed as in the past by an extension of the area of operations. Under present conditions it is unreasonable to expect a novel cure-all. For progress there is no cure. Any attempt to find automatically safe channels for the present explosive variety of progress must lead to frustration. The only safety possible is relative, and it lies in an intelligent exercise of day-to-day judgment \[…\]
> 
> The one solid fact is that the difficulties are due to an evolution that, while useful and constructive, is also dangerous. Can we produce the required adjustments with the necessary speed? The most hopeful answer is that the human species has been subjected to similar tests before and seems to have a congenital ability to come through, after varying amounts of trouble. To ask in advance for a complete recipe would be unreasonable. We can specify only the human qualities required:  
> patience, flexibility, intelligence.

This sounds an awful lot like what AI Alignment has been trying to do. Granted, as [Scott Alexander notes in a book review](https://astralcodexten.substack.com/p/book-review-the-man-from-the-future?utm_medium=email) of von Neumann's biography:

> This sounds suspiciously like the smartest man in the world admitting he’s not sure what to do.
# Any tips for eliciting one's own latent knowledge?

I struggle to write. I cannot figure out how to access my own thoughts and ideas. They come in non-linguistic forms - vague feelings, 3D images, animations - with a verbal narrator on top, but my very bad memory, and the fact that my best thinking is done in the middle of the night when I ought to be asleep, means that I end up forming intuitions about things based upon my prior thoughts, but I'm unable to remember what those thoughts were, or even what intuitions have recently formed how. Instead, when something reminds me of a topic, I remember whatever is salient that I've already thought of at some unknown prior time.

I don't possess a map of my own mind, nor even a search function. Just a mysterious black box. My mind is totally disorganized. And when I look at a blank page, I have no idea what to write, where to start. I don't even know what questions to ask myself to prompt writing, nor how to answer any. I honestly have *no idea how to think clearly*. Particularly from a blank slate, without prompting or real-time communication with another person, who can dig things out of me I cannot retrieve on my own, by reminding me of things with their questions.

I've heard of the concept of "solve the whole problem day", but alignment itself is not in my wheelhouse - I'm more of a "philosopher" (though that's much too pretentious to call myself), and my "problems" are vague and undefined - nebulous clouds of concepts on which I fixate, returning again and again, like the nature (or, phenomenology?) of identity, boundaries, emotions - or plans for a digital platform for increasing collective intelligence, rationality, and coordination. And my "plans" are composed of an endless array of disconnected fragments I cannot put together - because, again, specific details only show up in my mind temporarily like electrons falling out of superposition, and then they return to being a cloudy orbital of pure intuition.

Does anyone else know what this feels like and have any tips for me? It's *maddening*, and I've been suffering from it for basically my whole life. I've tried using the Zettelkasten system to "organize" my thoughts but it doesn't work for me, the result is just a tangled mess as bad as my real brain, no easier to parse through. I want to be of use to this site, providing interesting perspectives, particularly in the search for "True Names" and the design of support structures like the aforementioned vaguely imagined platform - but I can't do that if I can't actually pull info out of my head, organize it, and write it up for others to read.
# Deep learning curriculum for large language model alignment

This is a deep learning curriculum with a focus on topics relevant to large language model alignment. It is centered around papers and exercises, and is biased towards my own tastes.

It is targeted at newcomers to deep learning who are already familiar with the basics, but I expect it to be pretty challenging for most such people, especially without mentorship. So I have included some advice about prerequisites and more accessible alternatives.
# How to impress students with recent advances in ML?

I am preparing resources for an equivalent of the [mlab](https://www.lesswrong.com/posts/3ouxBRRzjxarTukMW/apply-to-the-second-iteration-of-the-ml-for-alignment) camp, organized by [EffiSciences](https://www.effisciences.org/),  that will be held in France in a month to train students to become prosaic alignment researchers.

3/4 of the participants are already aligned with existential risks related to AI, but 1/4 of the students have been selected for their technical skills and epistemologies, but are not yet familiar with the AI safety issue.

We have the intuition that one of the main reasons why they do not consider this issue is their lack of knowledge about the very recent capabilities of AI.

If you had 30 minutes to impress them with the advances in deep learning and the speed at which the field is progressing, what resources would you recommend? What should I present? How would you pitch this?
# Better Quiddler

[Quiddler](https://boardgamegeek.com/boardgame/339/quiddler) is a card game, a cross between rummy and scrabble. I've been playing it on vacation with in-laws, and it suffers from the common drawback of games where your turn begins with drawing and ends with discarding: you begin your turn with new information, and so have to do a lot of thinking while everyone waits for you. This is something more recent games [have fixed](https://boardgamegeek.com/boardgame/36218/dominion) and we can make the same fix to Quiddler, rotating the steps.

Current turn:

1.  Draw a card, from either the discard or draw piles
2.  Optionally put down words
3.  Discard a card

Better turn:

1.  Optionally put down words
2.  Discard a card
3.  Draw a card, from either the discard or draw piles

You each have your own personal discard pile to your right:

[![](https://www.jefftk.com/better-quiddler.jpg)](https://www.jefftk.com/better-quiddler-big.jpg)

The reason we need personal discard piles is that if you keep having a common one then drawing from it is nearly useless: the card on top will always be the one you just decided to dump. You could instead have the option to pick up the card the player before you discarded, which sort of fixes this, but then there's still important information you receive just as your turn starts. Best is to have the option to pick up the card discarded by the player who goes *after* you: that's the one that's been sitting out for the longest. Each player has their own discard pile to their right, and when you want to pick up a face-up card you draw from the discard pile to your left.

The other change we need to make is that at the beginning of each hand everyone choses between drawing a face up card or one from the draw pile. This step can be simultaneous. You could instead deal an extra card to everyone to begin, but getting a little bit of choice about what your initial hand contains reduces the chances of horrible (no vowels or all vowels) hands.
# Don't use 'infohazard' for collectively destructive info

Suppose there were a cheap way to make nuclear weapons out of common household materials. Knowing this information is not directly harmful to the median individual like yourself. On average, it may even be beneficial to the average person, to know that secret yourself; either because you could directly conquer places, or because you could sell the info to somebody who could. But society as a whole would rather that nobody knew it than that everybody knew it.

For a more realistic example, consider the DNA sequence for smallpox: I'd definitely rather that nobody knew it, than that people could with some effort find out, even though I don't expect being exposed to that quoted DNA information to harm my own thought processes.

'Infohazard', I think, should be used only to refer to the slightly stranger case of information such that the individual *themselves* would rather not hear it and would say, "Don't tell me that".

A spoiler for a book you're in the middle of reading is the classic example of information which is anomalously harmful to you personally. You might pay a penny not to hear it, if the ordinary course of events would expose you to it. People enclose info about the book's ending inside spoilers, if on Reddit or Discord, not because they mean to keep that secret to benefit themselves and harm you - as is the more usual case of secrecy, in human interactions - but because they're trying to protect you from info that would harm you to know.

The term 'infohazard' has taken on some strange and mystical connotations by being used to refer to 'individually hazardous info'. I am worried about using the same word to also refer to information being alleged to be dangerous in the much more mundane sense of 'collectively destructive info' - info that helps the average individual hearer, but has net negative externalities if lots of people know it.

This was originally posted by me to Facebook at greater length (linkpost goes to original essay).  Best suggestions there (imo) were 'malinfo' and 'sociohazard', and if I had to pick one of those I'd pick 'sociohazard'.

EDIT:  I like shminux's 'outfohazard' even better.

EDIT 2:  My mind seems to automatically edit this to 'exfohazard' so I guess that's what the word should be.
# Peter Singer's first published piece on AI 

Hi! I am Tse Yip Fai. This is a linkpost to Peter Singer's first published piece on AI, it's also my first [published paper](https://link.springer.com/article/10.1007/s43681-022-00187-z) (second author). Since the paper is open access, I won't be doing any copy-and-paste here. 

This is a seminal paper resulting from an one-year research. We hope that this paper can break the ground and raise awareness in the field of AI ethics so that they start caring about the impact on animals. This, I believe, is not just relevant to the AI ethics field, but also to the AI alignment field. 

If you prefer videos, [here](https://mediacentral.princeton.edu/media/CITP%20Seminar%20Peter%20Singer%20and%20Yip%20Fai%20Tse%20-%20AI%20Ethics%3A%20The%20Case%20for%20Including%20Animals/1_b1b7bqq8)'s a talk by Professor Singer and myself on the same topic.
# Why I Think Abrupt AI Takeoff


*(note: Quickly written. I've attempted to number my arguments "formally", but I have no training in this format. Edits/suggestions welcome.)*

1. Ability to generalize is "lumpy": unpredictable -- sometimes lots of inputs lead to little progress, sometimes small inputs lead to lots of progress.
2. Takeoff is threshold-based: When machines get enough ability to generalize, takeoff will happen.

*thus*

3. As we increase the ability of machines to generalize, we have a high chance of putting in a small amount of input and getting over the takeoff threshold.

---

Argument for claim 1:

4. Lumpy input->output curves exist in individual domains as a result of forming new and powerful abstractions in those domains.
5. Cognitive architecture components generalize across domains as in (4).
6. Cognitive architecture is itself a "domain" as in (4) and thus lumpy.  

*thus*

1. *(restated)* If we get a new abstraction in the form of a cognitive architecture component that generalizes across domains, we will see rapid cross-domain progress, and thus **ability to generalize is lumpy.**

---

Evidence for claim 4 — "Narrow" lumpy generality — specifically, new and powerful abstraction gives rise to abrupt domain-specific performance improvements:

7. Anecdotally, humans have a lumpy learning experience. It is common for people to talk about "Aha moments", "eureka moments" and so on. 
8. AlphaGo Zero seems somehow relevant (although I have trouble pointing at precisely how, but it convinces me regardless)
9. [The "Grokking" paper](https://arxiv.org/pdf/2201.02177.pdf) (thanks to Quintin Pope and other commenters!)

---

Evidence for claim 5 — cognitive architecture components generalize:

10. Chimps -> humans: Humans' cognitive architecture is very general and allows us to quickly form a broad set of domain-specific abstractions that perform extremely well with little energy.

---

Evidence for claim 2 — takeoff is threshold-based:

11. Chimps -> humans: chimps didn't take over the world but humans did.
12. (The usual arguments for hardware overhang, ability to scale and copy what works, etc.)
# Four questions I ask AI safety researchers

Over the last few months, I’ve been trying to develop a stronger inside view on AI safety research agendas. 

As part of this quest, I’ve been having conversations with AI safety researchers. I notice myself often asking the following questions:

1.  What are you working on, and how does it help us get aligned AI?
2.  Imagine you’re talking to a smart high school student. How would you describe the alignment problem? And how would you describe how your work addresses it?
3.  Imagine that you came up with a solution to the specific problem you’re working on. Or even more boldly, imagine your entire program of research succeeds. What happens next? Concretely, how does this help us get aligned AI (and prevent unaligned AI)?
4.  What are the qualities you look for in promising AI safety researchers? (beyond general intelligence)

I find Question #1 useful for starting the conversation and introducing me to the person’s worldview.

I find Question #2 useful for getting clearer (and often more detailed) explanations of the person’s understanding of the alignment problem & how their work fits in. (*Note that this is somewhat redundant with question #1, but I find that the questions often yield different answers*).

I find Question #3 useful for starting to develop an inside view on the research agenda & the person’s reasoning abilities. (*Note that the point is to see how the person reasons about the question. I’m not looking for a “right answer”, but I am looking to see that someone has seriously thought about this and has reasonable takes.) *

I find Question #4 useful for building stronger models of AI safety community-building.

I also ask a lot of follow-up questions. But I find that these are the four questions that I ask nearly everyone, and I think they’ve helped me develop a stronger inside view.

If I had to pick one question, I would probably ask #3. I think it’s a rather challenging question, and I’m generally impressed when people have coherent answers.
# LessWrong Meetup - Gendlin's Focusing

Focusing has long been part of the rationalist toolbox, since it can be seen as a technique to bridge "gut feelings" (system I) and "thinking" (system II), a way to more clearly access knowledge we already have, and possibly fine tune it.

The basics are quite simple, and in this workshop we will experiment with the basic "motions" of Focusing  
accessing felt sense, listening to it, feeling "inner motions", giving a name to a sensation.

We will move from "I have it on the tip on my tongue" to "it is almost as if... but not really" to "yeah, exactly like that", and find ways to support each other via listening and mirroring.

Some introductory articles:  
https://www.lesswrong.com/posts/f3o9ydY7iPjFF2fyk/focusing-1  
https://www.lesswrong.com/posts/PXqQhYEdbdAYCp88m/focusing-for-skeptics  
https://www.lesswrong.com/posts/eccTPEonRe4BAvNpD/the-felt-sense-what-why-and-how

I invited Alessandro to give this workshop as he has a lot of experience with Focusing outside of the usual rationalist sphere. Outside of that, he spends part of his life trying to tell quite primitive AIs what to do (at times successfully), and another part collecting techniques and ideas dealing with how human work together, connect with others, and themselves. Among the rest, he organizes a semi-regular meetup called "the cookbook" (think about an unconference, in a 90' box, about life and relationship skills), facilitates workshops, and co-hosts retreats.  
You can find more here:  
https://wetwarecraft.com/about/who-am-i/

14:00-14:15 Arriving and chatting  
14:15-15:15 First Workshop Block  
15:15-15:45 Mingling  
15:45-17:15 Second Workshop Block  
17:15-18:00 Mingling

Given that we are outdoors it's useful to bring a blanket along. You can find the location either at [https://w3w.co/fühlt.beugen.freudig](https://w3w.co/f%C3%BChlt.beugen.freudig) or via [meetup](https://www.meetup.com/lesswrong-rationality-waitbutwhy-slatestarcodex-berlin/events/287286525/). Given that it's a public park we might be located at a slightly different spot if another group already occupies the spot before we arrive.
# Sexual Abuse attitudes might be infohazardous

*Content warning: sexual abuse, rape, arguably trivialising thereof.*

---

When I was 11 an older boy used to pull me behind a door in one of the school corridors, trap me there, shove his hand down my pants, and touch my penis.

This happened about once a week for a couple of months, until I moved to a different school (for non-related reasons).

I knew it was taboo, but I didn't yet know about sex, and didn't really understand why. I never thought about it again till a few years later when he sent me a letter apologizing. My parents were extremely curious about what the letter was but I burnt it before they had a chance to read it.

I never told anyone about this until now, and even now I'm writing this pseudonymously.

Not because I'm ashamed or embarrassed. I'm not - I didn't do anything to be ashamed of. But because I'm fine.

I really am. I don't think this made any lasting impact to my life. I'm happy, well adjusted, married, successful etc. I just don't really think about what happened very much, but then again I rarely think about anything that happened to me when I was 11.

And yet I feel like society is telling me that I ought to be broken. That I've been sexual abused. That recovering from this will be a difficult painful process, probably requiring therapy. And I fear that if I tell someone that, they'll treat me like that's the case, and I might end up believing it about myself.

---

From a purely objective perspective, non-violent rape doesn't seem quite as bad as society makes it out to be.

It's obviously unpleasant and frightening, but we treat rape as one of the worst things that can possibly happen. We expect "rape victim" to become someone's whole identity. We expect them to need intensive therapy to put themselves back together.

And I'm sure for plenty of people that's true. But for plenty of others it's true only because we expect it of them. People fill the social role that's been made for them, even when that's not ideal for them.

I don't know what to do about this. How do we communicate that sexual abuse is really not ok, without making victims of it feel like it's worse than it actually is?

But at the very least, when you hear somebody's abuse story don't jump straight into treating them like a victim. Find out how they feel about it, and if they don't feel like it was that bad, there's really no need for you to make them feel like it was worse.
# Progress links and tweets, 2022-07-19

**Links**
---------

*   [The first images from the JWST look amazing](https://www.nasa.gov/webbfirstimages). See also [this comparison with Hubble](https://twitter.com/ianlauerastro/status/1546625603641790465)
*   [Helion Energy is hiring. “Clean, continuous energy for 1 cent per kilowatt-hour”](https://blog.samaltman.com/helion-needs-you)
*   [Lant Pritchett on the difference between anti-poverty programs and poverty reduction](https://lantpritchett.org/development-work-versus-charity-work/). To paraphrase Rutherford, all poverty reduction is either economic growth or stamp collecting
*   [Joel Mokyr reviews Koyama & Rubin’s *How the World Became Rich*](https://eh.net/book_reviews/how-the-world-became-rich-the-historical-origins-of-economic-growth/)
*   [Potential for a vaccine against a broad category of SARS-like coronaviruses](https://www.science.org/doi/10.1126/science.abq0839) (in *Science*; see also the [announcement from Wellcome Leap](https://wellcomeleap.org/news_mosaic-8/) which funded this)

**Threads**
-----------

*   [Many waste byproducts were developed in Britain during the Industrial Revolution](https://twitter.com/VincentGeloso/status/1546863018172194816) (@VincentGeloso). See also: [In capitalism they use every part of the animal](https://twitter.com/jasoncrawford/status/1430389158942580736)
*   [“Reclosers” on electrical utility poles](https://twitter.com/HillhouseGrady/status/1546860065940029441) (@HillhouseGrady)

**Queries**
-----------

*   [Why is hating humanity acceptable?](https://twitter.com/CineraVerinia/status/1546791669927972864) (@CineraVerinia)
*   [Why doesn’t washing hands create hand-washing resistant bacteria?](https://twitter.com/_brianpotter/status/1546925218949742597) (@_brianpotter)

**Tweets**
----------

*   [Request for blog: the history of social change, and lessons for modern movements](https://twitter.com/jasoncrawford/status/1546566380685778944)
*   [Eradicating smallpox sometimes meant breaking the rules](https://twitter.com/jasoncrawford/status/1546999169209999368)
*   [India might already the most populous country in the world](https://twitter.com/jasoncrawford/status/1546558934324068354)

**Retweets**
------------

*   [Terraform Industries can convert H~2~ and CO~2~ into natural gas](https://twitter.com/TerraformIndies/status/1547828697427628032) (@TerraformIndies)
*   [Montessori on the need to teach “the highest discoveries” of science](https://twitter.com/mbateman/status/1546387186890719233) (@mbateman)
*   [There’s an Industrial Revolution board game set in 18th-c. Lancashire](https://twitter.com/dkedrosky/status/1548340636545232896) (@dkedrosky)

[![](https://pbs.twimg.com/media/FXzPZt6UEAIlDRW.jpg)](https://pbs.twimg.com/media/FXzPZt6UEAIlDRW.jpg)
# A daily routine I do for my AI safety research work

I’m a Ph.D student at MIT working in AI safety--mostly interpretability in deep networks. I thought I would very briefly share a list of links that I have bookmarked and go through every weekday. It usually takes me <15 minutes. Since I started doing this, I’ve been much more up to date and informed on current work. 

First is [Twitter](https://twitter.com/). For a long time, I resisted using Twitter because it’s terrible. But after I started following almost exclusively AI safety people and orgs, I’ve found a lot of good papers and ideas. 

Second, I check [LessWrong](https://www.lesswrong.com/) and the [AI Alignment Forum](https://www.alignmentforum.org/) :) 

Finally, I look over new titles on arXiv for [AI](https://arxiv.org/list/cs.AI/recent), [ML](https://arxiv.org/list/cs.LG/recent), [Computation and Language](https://arxiv.org/list/cs.CL/recent), and [Computer Vision and Pattern Recognition](https://arxiv.org/list/cs.CV/recent). More often than not, I find a new paper that I want to click on and check out. 

And as a bonus, I’d of course also recommend [these](https://rohinshah.com/alignment-newsletter/) [three](https://jack-clark.net/) [newsletters](https://newsletter.mlsafety.org/).
# What are the simplest questions in applied rationality where you don't know the answer to? 

In the podcast between Spencer Greenberg and Buck Shlegeris, [Taking pleasure in being wrong](https://open.spotify.com/episode/4Q54moES6YvjCEyOORRgX3), Buck says:

I think that when you are learning subjects something that you should really be taken your eye out for is the simplest question in the field that you don't know the answer to.

> I think a lot of the time people try to learn physics or something and their approach is as quickly as possible to answer hard questions about complicated subjects. And I think that's what I thought was cool when I was younger. They delighted at questions that were at the limit of fanciness that they could possibly answer and it feels to me now that it is a lot more productive to seek out questions that are as simple sounding as possible while still being really hard to answer. Or that still demonstrate that there's something you don't understand about the subject.
> 
> \[...\] 
> 
> It seems like we should be seeking out these most basic questions in the hope of finding holes in the foundation of our knowledge.

If we apply that approach to applied rationality, what questions do you have that seem to be simple but where you don't know the answer?
# Is Gas Green?

The EU decided to declare gas along with nuclear to be[ green](https://www.dw.com/en/european-commission-declares-nuclear-and-gas-to-be-green/a-60614990). It's easy to see why one might see nuclear energy as a green energy source, it's harder to see the case for gas. 

As I[ previously pointed out](https://www.lesswrong.com/posts/QetNqj7Dnrt5dcaGN/what-will-we-do-with-the-free-energy), we currently have days in Germany where energy is free and companies can get paid to use up energy. The continuously lowering cost of solar power suggests that those days will increase. Rolling out enough solar and wind to be able to power everything on most days means massive overproduction in electricity on summer days that can be used for applications that need a lot of cheap energy but that are okay with only running when energy production is high. One option for using the energy is to turn [CO2 into Synthetic Natural Gas](https://www.sciencedirect.com/science/article/pii/S0016236119321581) (SNG):

The overall efficiency of energy storage and reuse (power-to-power) is quite high. In the case of SNG synthesis, the energy stored in methane accounts for 53.2% of the input energy. The maximum recoverable energy in the form of heat and electricity is 37.3%

If we predict a future where we will use surplus solar energy to produce SNG, it today makes sense to build gas plants to provide us with energy on winter days with little wind. It’s building plants that will actually provide green energy in the future, which justifies seeing them as green in the EU energy plan.

It also suggests that moving a huge sector like shipping to using natural gas is a valid strategy for making the sector completely green.
# How much to optimize for the short-timelines scenario?

Some have argued that one should tend to act as if timelines are short since in that scenario it's possible to have more expected impact. But I haven't seen a thorough analysis of this argument.

**Question:** Is this argument valid and if yes how strong is it?

The basic argument seems to be: if timelines are short, the field (AI alignment) will be relatively smaller and have made less progress. So there will be more low-hanging fruits and so you have more impact.

The question affects career decisions. For example, if you optimize for long timelines, you can invest more time into yourself and delay your impact.

The question interacts with the following questions in somewhat unclear ways:

*   How fast do returns to more work diminish (or increase)?
    *   If returns don't diminish, the argument above fails. 
    *   If the field will grow very quickly, returns will diminish faster. 
*   Is your work much more effective when it's early? 
    *   This may happen because work can be hard to parallelize - ‘9 women can't make a baby in 1 month’. And field-building can be more effective earlier as the field  can compound-grow over time. So someone should start early. 
    *   If work is most effective earlier, you shouldn’t lose too much time investing in yourself. 
*   Is work much more effective at crunch time? 
    *   If yes, you should focus more on investing in yourself (or do field-building for crunch time) instead of doing preparatory research. 
*   If timelines are longer, is this evidence that we'll need a paradigm shift in ML that makes alignment easier/harder? 
    *   (This question seems less tractable than the others.) 
*   Is your comparative advantage to optimize for short or long timelines? 
    *   For example, young people can contribute more easily given longer timelines and vice versa. 

If someone would like to seriously research the overall question, please reach out. The right candidate can get funding.