Harley-ml/lesswrong huggingface dataset:
# Don't Condition on no Catastrophes

I often hear people say things like "By what date do you assign 50% chance to reaching AGI, conditioned on no other form of civilizational collapse happening first?" The purpose of this post is to make this question make you cringe.

I think that most people mentally replace the conditional with something like "if other forms of civilizational collapse were magically not a thing, and did not have to enter into your model." Further, I think this is the more useful question to discuss, as it makes it easier to double crux, or download other people's models. However, not everyone does this, and it is not the question being asked.

To illustrate the difference, consider Alice, who, if they ignored other civilizational collapse, would think that AGI arrival date is uniform over the next 100 years. However, they also think that if not for AGI, extinction level nuclear war will happen in the next 100 years, uniformly at random over the next 100 years. Alice is not concerned about any other catastrophes.

Alice has these two independent distributions on when each event will happen if counterfactually, the other were to magically not happen. However, the world is such that as soon as one of events happens, it causes the other event to not happen, because the world is made very different.

When asking about Alice's median AGI date, ignoring civilizational collapse, we would like to encourage her to say 50 years. However her median AGI date, conditional on no nuclear war happening first is actually 33 years. This is because conditioning on no nuclear war happening first biases towards AGI dates that are early enough to stop a counterfactual future nuclear war.

The form of the question I would like to ask Alice is as follows:

Take your distribution over ways the way the future can go, and sample a random future, < refresh to render LaTeX > . If that future ends with nuclear war at time < refresh to render LaTeX > , sample another world with the property that neither AGI nor any other catastrophe happens before time < refresh to render LaTeX > . If that world ends with a non AGI catastrophe, redefine < refresh to render LaTeX > to be the time of the catastrophe in that world, and repeat the process, until you get a world that ends with AGI, with no other catastrophe happening first. Use this as your new distribution over futures, and tell me the median AGI date.

Note that conditioning on no other catastrophe happening first is the same procedure, except when you sample a new future, you do not require that it has the property that neither AGI nor any other catastrophe happens before time < refresh to render LaTeX > .

I don't have a good name for this alternative to conditioning, and would like suggestions in comments. You may notice a similarity between it and causal counterfactuals. You also notice a similarity between it and the thing you do to get Solomonoff Induction out of the Universal Semimeasure.
# Robustness to Scale

I want to quickly draw attention to a concept in AI alignment: Robustness to Scale. Briefly, you want your proposal for an AI to be robust (or at least fail gracefully) to changes in its level of capabilities. I discuss three different types of robustness to scale: robustness to scaling up, robustness to scaling down, and robustness to relative scale.

The purpose of this post is to communicate, not to persuade. It may be that we want to bite the bullet of the strongest form of robustness to scale, and build an AGI that is simply not robust to scale, but if we do, we should at least realize that we are doing that.

Robustness to scaling up means that your AI system does not depend on not being too powerful. One way to check for this is to think about what would happen if the thing that the AI is optimizing for were actually maximized. One example of failure of robustness to scaling up is when you expect an AI to accomplish a task in a specific way, but it becomes smart enough to find new creative ways to accomplish the task that you did not think of, and these new creative ways are disastrous. Another example is when you make an AI that is incentivized to do one thing, but you add restrictions that make it so that the best way to accomplish that thing has a side effect that you like. When you scale the AI up, it finds a way around your restrictions.

Robustness to scaling down means that your AI system does not depend on being sufficiently powerful. You can't really make your system still work when it scales down, but you can maybe make sure it fails gracefully. For example, imagine you had a system that was trying to predict humans, and use these predictions to figure out what to do. When scaled up all the way, the predictions of humans are completely accurate, and it will only take actions that the predicted humans would approve of. If you scale down the capabilities, your system may predict the humans incorrectly. These errors may multiply as you stack many predicted humans together, and the system can end up optimizing for some seeming random goal.

Robustness to relative scale means that your AI system does not depend on any subsystems being similarly powerful to each other. This is most easy to see in systems that depend on adversarial subsystems. If part of you AI system is suggest plans, and another part is trying to find problems in those plans, if you scale up the suggester relative to the verifier, the suggester may find plans that are optimized for taking advantage of the verifier's weaknesses.

My current state is that when I hear proposals for AI alignment that do not feel very strongly robust to scale, I become very worried about the plan. Part of this comes from feeling like we are actually very early on a logistic capabilities curve. I thus expect that as we scale up capabilities, we can get eventually get large differences very quickly. Thus, I expect that the scaled up (and partially scaled up) versions to actually happen. However, robustness to scale is very difficult, so it may be that we have to depend on systems that are not very robust, and be careful not to push them too far.
# The map has gears. They don't always turn.

_Follow up to [Toward a New Technical Explanation of Technical Explanation](https://www.lesserwrong.com/posts/tKwJQbo6SfWF2ifKh/toward-a-new-technical-explanation-of-technical-explanation)._

[Confusion is in the map](https://wiki.lesswrong.com/wiki/Reality_is_normal), not the territory. The same goes for predictability, surprise, mysteriousness, weirdness, and so on. Yet, sometimes, we say "no, actually, that's quite surprising" after initially not being surprised -- and it has a meaning. Or, sometimes we say "ah, right, it _is_ obvious" after being confused and needing to think for a while -- and it has meaning.

Though they may be phrased as map/territory errors, statements like this can refer to the amount of surprise or confusion etc which a specific gears-level model has. A step in a proof may be "obvious" because it is an elementary application of a common method, even if it takes time for a fallible human to see it. A physicist may one day come to realize that a commonplace event is "surprising" after being not-surprised by the event dozens of times, because suddenly they've compared it to a model from physics, and found the event difficult to account for.

Our brain has a big messy model of reality, but it is often trying to simulate smaller, cleaner, more justified, more objective, and hopefully better models. When we do things like this, we are speaking from those models.
# The abruptness of nuclear weapons

Nuclear weapons seem like the marquee example of rapid technological change after crossing a critical threshold.

Looking at the numbers, it seems to me like:

*   During WWII, and probably for several years after the war, the cost / TNT equivalent for manufacturing nuclear weapons was comparable to the cost of conventional explosives, (A[I impacts](https://aiimpacts.org/discontinuity-from-nuclear-weapons/) estimates a manufacturing cost of $25M/each)
*   Amortizing out the cost of the Manhattan project, dropping all nuclear weapons produced in WWII would be cost-competitive with traditional firebombing (which [this thesis](https://ses.library.usyd.edu.au/bitstream/2123/664/2/adt-NU20050104.11440202whole.pdf) estimates at 5k GBP (=$10k?) / death, vs. ~100k deaths per nuclear weapon) and by 1950, when stockpiles had gown to >100 weapons, was an order of magnitude cheaper. (Nuclear weapons are much easier to deliver, and at that point the development cost was comparable to manufacturing cost).

Separately, it seems like a 4 year lead in nuclear weapons would represent a decisive strategic advantage, which is much shorter than any other technology. My best guess is that a 2 year lead wouldn't do it, but I'd love to hear an assessment of the situation from someone who understands the relevant history/technology better than I do.

So my understanding is: it takes about 4 years to make nuclear weapons and another 4 years for them to substantially overtake conventional explosives (against a 20 year doubling time for the broader economy). Having a 4 year lead corresponds to a decisive strategic advantage.

Does that understanding seem roughly right? What's most wrong or suspect? I don't expect want to do a detailed investigation since this is pretty tangential to my interests, but the example is in the back of my mind slightly influencing my views about AI, and so I'd like it to be roughly accurate or tagged as inaccurate. Likely errors: (a) you can get a decisive strategic advantage with a smaller lead, (b) cost-effectiveness improved more rapidly after the war than I'm imagining, or (c) those numbers are totally wrong for one reason or another.

I think the arguments for a nuclear discontinuity are really strong, much stronger than any other technology. Physics fundamentally has a discrete list of kinds of potential energy, which have different characteristic densities, with a huge gap between chemical and nuclear energy densities. And the dynamics of war are quite sensitive to energy density (nuclear power doesn't seem to have been a major discontinuity). And the dynamics of nuclear chain reactions predictably make it hard for nuclear weapons to be "worse" in any way other than being more expensive (you can't really make them cheaper by making them weaker or less reliable). So the [continuous progress narrative](https://sideways-view.com/2018/02/24/takeoff-speeds/) isn't making a strong prediction about this case.

(Of course, progress in nuclear weapons involves large-scale manufacturing. Today the economy grows at roughly the same rate as in 1945, but information technology can change much more rapidly.)
# Self-regulation of safety in AI research

In many industries, but especially those with a potentially adversarial relationship to society like advertising and arms, self regulatory organizations (SROs) exist to provide voluntary regulation of actors in those industries to assure society of their good intentions. For example:

*   TrustArc (formerly TRUSTe) has long provided voluntary certification services to web companies to help them assure the public that companies are following basic practices that allow consumers to protect their privacy. They have been successful enough to, outside the EU, keep governments from much regulating online businesses.
*   The US Green Building Council offers multiple levels of LEED certification to provide both targets and proof to the public that real estate developers are protecting environmental commons.
*   The American Medical Association, The American Bar Association, and the National Association of Realtors are SROs that function as de facto official regulators of their industries despite being non-governmental organizations (NGOs).
*   Financial regulation was formerly and sometimes still is done via SROs, although governments have progressively taken a stronger hand in the industry over the last 100 years.

AI, especially AGI, is an area where there are many incentives to violate societal preferences and damage the commons and it is currently unregulated except where it comes into contact with existing regulations in its areas of application. Consequently, there may be reason to form an AGI SRO. Some reasons in favor:

*   An SRO could offer certification of safety and alignment efforts being taken by AGI researchers.
*   An SRO may be well positioned to reduce the risk of an AI race by coordinating efforts that would otherwise result in competition.
*   An SRO could encourage AI safety in industry and academia while being politically neutral (not tied to a single university or company).
*   An SRO may allow AI safety experts to manage the industry rather than letting it fall to other actors who may be less qualified or have different concerns that do not as strongly include prevention of existential risks.
*   An SRO could act as a "clearinghouse" for AI safety research funding.
*   An SRO could give greater legitimacy to prioritizing AI safety efforts among capabilities researchers.

Some reasons against:

*   An SRO might form a de facto "guild" and keep out qualified researchers.
*   An SRO could create the appearance that more is being done than really is.
*   An SRO could relatedly promote the wrong incentives and actually result in less safe AI.
*   An SRO might divert funding and effort from technical research in AI safety.

I'm just begining to consider the idea of assembling an SRO for AI safety, and especially interested in discussing the idea further to see if it's worth pursuing. Feedback very welcome!
# Will AI See Sudden Progress?

Will advanced AI let some small group of people or AI systems take over the world?

AI X-risk folks and others have accrued lots of arguments about this over the years, but I think this debate has been disappointing in terms of anyone changing anyone else’s mind, or much being resolved. I still have hopes for sorting this out though, and I thought a written summary of the evidence we have so far (which often seems to live in personal conversations) would be a good start, for me at least.

To that end, I started a **[collection of reasons to expect discontinuous progress near the development of AGI](http://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/)**.

I do think the world could be taken over without a step change in anything, but it seems less likely, and we can talk about the arguments around that another time.

Paul Christiano had basically the same idea at the same time, so for a slightly different take, **[here](https://sideways-view.com/2018/02/24/takeoff-speeds/)** is his account of reasons to expect slow or fast take-off.

Please tell us in the comments or **[feedback box](https://aiimpacts.org/feedback/)** if your favorite argument for AI Foom is missing, or isn’t represented well. Or if you want to represent it well yourself in the form of a short essay, and send it to me **[here](mailto:katja@intelligence.org)**, and we will gladly consider posting it as a guest blog post.

I’m also pretty curious to hear which arguments people actually find compelling, even if they are already listed. I don’t actually find any of the ones I have that compelling yet, and I think a lot of people who have thought about it do expect ‘local takeoff’ with at least substantial probability, so I am probably missing things.

* * *

_Crossposted from [AI Impacts](https://aiimpacts.org/will-ai-see-sudden-progress/)._
# Experimental Open Threads

[Meta-tations on Moderation: Towards Public Archipelago](https://www.lesserwrong.com/posts/5Ym7DN6h877eyaCnT/meta-tations-on-moderation-towards-public-archipelago) suggested that we should be running more experiments with how discussion is run and listed several examples in the post. What if we ran monthly experimental open threads? I can imagine several formats that would work better as open threads than as a thread belonging to a specific commentator.

Here's a few thoughts of possible experiments:

*   Double crux - Top level comment would be claims. People who disagree can comment underneath to double crux with the poster. Third parties are discouraged from adding comments which inject their opinion into the double crux
*   Change my view - Top level comments would be claims that the author must believe and be willing to defend. Second level comments must disagree with at least one point of the claim or ask for clarification
*   Socratic method - Top level comments would be claims. Second level comments would be discouraged from directly saying that someone is wrong and instead encouraged to ask them questions instead to get them to think
*   Explain your perspective - Very similar to Socratic method, but the Socratic method is still focused on persuasion. This would also involve asking questions, but the commentators would be encouraged to try to understand people's views instead of trying to share them
*   Dependency sharing - Top level comments list a view and give examples of what might convince you to
*   Ask me anything - Top level comments lists an experience someone has had or specific expertise. Lower level comments ask them questions

A key part of rationality that has been somewhat underemphasised on Less Wrong is group rationality. Achieving big goals requires groups of people and these people need to be able to co-ordinate and efficiently share their knowledge. We could sit around and try to theorise about how to make this work or we could simply start running experiments.

A few questions:

*   Would these be suitable for frontpage posts? I don't see this experiment working with blog posts, as they would be unlikely to hit critical mass. (Sidenote: I don't think posting these threads should net me karma, if they take off it may be worthwhile to have a way to mark threads as open at some point)
*   Any suggestions of useful experiments to run?
*   Lastly, if I ran some of these experiments, would anyone commit to participating significantly in at least the first N (default N=3) to help achieve this critical mass? By participating significantly I mean making a top level comment, addressing replies to this comment and participating in at least one other sub-thread.
# Mapping the Archipelago

I got excited reading [Meta-tations on Moderation: Towards Public Archipelago](https://www.lesserwrong.com/posts/5Ym7DN6h877eyaCnT/meta-tations-on-moderation-towards-public-archipelago) for two reasons: there's a clear island of the archipelago I've been mostly avoiding on LessWrong, and the whole place has been growing at a high enough rate to demand fracturing.

Since we have the chance to direct the growth of the brand new archipelago, let's start a discussion down one level of meta: what specific islands do you want to see? Second, how should discussion and moderation norms differ between them?

Three islands of current LW, according to me:

1.  AI Risk: Serious discussion for serious folk. No smiles allowed.
2.  Instrumental Rationality: The means to get to the ends. Whatever they might be.
3.  Fluff and Fiction: Blood for the Art God! Fun over fact.

(Guess which island I avoid.)
# Inconvenience Is Qualitatively Bad

My most complicated cookie recipe has four layers. Two of these require stovetop cooking, and the other two require the use of the oven separately before the nearly-complete cookies are baked in yet a third oven use, for a total of three different oven temperatures. I have to separate eggs. I have to remember to put butter out hours in advance so it'll be softened when I get underway. Spreading the fruit neatly and then the almond goop on top of that without muddling the layers is finicky and almost none of the steps parallelize well.

They're delicious, but at what cost?

People who don't cook as a hobby would never, ever make these cookies. And this is reasonable. They shouldn't. On most days I shouldn't either. They are staggeringly inconvenient.

But they're made up of individual steps that you could mostly figure out if you really wanted to. Lots and lots of little steps. This is why I want to scream whenever I hear someone try to _add steps_ to someone else's life. Especially if they say "just".

"Just" Google it. "Just" rinse out your recyclables. "Just" add another thing to remember and another transition to your to-do list and another obligation to feel guilt about neglecting and another source of friction between you and your real priorities. It "just" takes a minute. Don't you _care_?

Anyone who didn't have any immune defense against things that just take a minute would spend fifteen hours a day on one-minute tasks, if expending the energy required to switch between the tasks didn't kill them before it got that bad. But "it would be inconvenient" doesn't tend to feel like a solid rebuttal - to either party; the one attempting to impose can just reiterate "but it'll only take a _minute_".

Everyone needs algorithms to cut down on inconveniences.

Some I am aware of:

*   Chunking. Things feel less inconvenient (and accordingly are) if they are understood in batches, as one thing and not thirty. (This is related, I think, to a lot of manifestations of executive dysfunction - chunking doesn't work as well.) People naturally do more of this with things they're good at - I think a lot of being good at things _just is_ being able to take them in larger chunks and finding larger amounts of the thing trivial.
*   Delegating. For some people delegating is itself inconvenient but if you delegate enough things it can be useful on balance. Often costly in other ways too - quality, customization, money.
*   Straight-up rejecting impositions, in whole ("I just won't recycle") or in part ("I'll recycle but no way am I washing out bean cans"). Pick what to reject at whim, from certain sources, or by another discrimination mechanism. Rejecting impositions from interactive humans as opposed to generic announcements or oneself requires social grace or a willingness to do without it.
# Goal Factoring

This is part 18 of 30 of Hammertime. Click [here](https://radimentary.wordpress.com/2018/01/29/hammertime-day-1-bug-hunt/) for the intro.

Up until today, Hammertime focused on improving one’s ability to achieve one’s goals. The next two techniques, Goal Factoring and Internal Double Crux, are designed to figure out what goals to pursue. For the largest goals in life, you should be able to make a detached decision about whether they’re worth pursuing before you throw your all into them.

Day 18: Goal Factoring
======================

Previously: [Day 7, Aversion Factoring](https://radimentary.wordpress.com/2018/02/04/hammertime-day-7-aversion-factoring/).

Goal Factoring is a CFAR technique for systematically figuring out all the subgoals and aversions you have around an action, and what to do about them. The basic algorithm:

1.  Pick an action. It can be something you already do.
2.  Factor the action into goals and aversions. Write down all the costs and aversions to pursuing the action, and continue to factor sub-goals until they feel like irreducible components.
3.  Brainstorm possible replacement actions. Try to design another action that achieves the goals better and reduces the costs and aversions. This action can be an upgrade of your current action, something else altogether, or even a combination of two or more actions. Make a new plan.
4.  Reality check. Imagine instituting your new plan. Decide if you’re satisfied. Also, decide if it’s feasible by [Murphyjitsu](https://radimentary.wordpress.com/2018/02/07/hammertime-day-10-murphyjitsu/).

This is already quite a complicated and useful beast. Three things to keep in mind:

Use a [focusing check](https://radimentary.wordpress.com/2018/02/25/focusing/) to find all the subgoals and aversions. If I say out loud, “The only reason I want to go to the gym is physical health,” I feel a curtain of discontent that reminds me physical attractiveness is also important. Remember that honesty and attention to detail are essential to finding aversions, and this applies to goals as well!

Goal factoring might solve the problem at any step. Writing down your true motivations can be enough to figure out the right course of action. About three months ago, I noticed that the main motivation for my video game addiction is “Prove to my parents that it’s possible to be successful without giving up video games.” Writing this down made it impossible to endorse this action any longer.

Prepare to accept all possible worlds. Keep an open mind going into Goal Factoring: you’re allowed to consider all the alternatives. You’re also allowed to keep doing what you’re currently doing afterwards. Try to release any attachment to the action itself beyond its instrumental value. Get a little worried if your main reason to do action X is to become the kind of person who does X, but at least write this down as an explicit sub-goal.

Exercise: pick an action or habit you want to pick up or drop, and set a Yoda Timer for 20 minutes to Goal Factor it.

Daily Challenge
===============

Set a Yoda Timer to Goal Factor “do Hammertime.” Share your motivations and aversions.
# Intuition should be applied at the lowest possible level

Earlier today I lost a match at Prismata, a turn-based strategy game without RNG. When I analyzed the game, I discovered that changing one particular decision I had made on one turn from A to B caused me to win comfortably. A and B had seemed very close to me at the time, and even after knowing for a fact that B was far superior, it wasn't intuitive why.

Then I listed the main results from A and B, valued those by intuition, and immediately B looked way better.

One can model these problems on a bunch of different levels, where going from level n to n+1 means hiding the details of level n and approximating their results in a cruder way. On level 1, one would compare the two subtrees whose roots are decisions A and B (this should work just like in chess). Level 2 would be looking at exact resource and attack numbers in subsequent turns. Level 3 would be categorizing the main differences of A and B and giving them intuitive values, and level 4 deciding between A and B directly. What my mistake showcases is that, even in a context where I am quite skilled and which has limited complexity, applying intuition at level 4 instead of 3 lead to a catastrophic error.

If you can't go lower, fine. But there are countless cases of people using intuition on a level that's unnecessarily high. Hence [if it's worth doing, it's worth doing with made-up numbers](http://slatestarcodex.com/2013/05/02/if-its-worth-doing-its-worth-doing-with-made-up-statistics/). That is just one example of where applying intuition one level further down: "what quantity of damage arises from this" rather than "how bad is it" can make a big difference. On questions of medium importance, briefly asking yourself "is there any point where I apply intuition on a level that's higher than necessary" seems like a worthy exercise.

Meta: I write this in the spirit of [valuing obvious advice](http://mindingourway.com/obvious-advice/), and the suspicion that this error is still made fairly often.
# 2/27/08 Update – Frontpage 3.0

We finally got around to a long-postponed update to the frontpage, where logged in users see content that is hopefully more relevant to them.

Some notes:

*   Non logged in users should have a mostly unchanged experience
*   Logged in users now see a list of recommended sequences. The list will change over time. We aren't sure about the details, but we want to highlight old content that might be worth re-reading, and new sequences that seem high caliber. For now we expect to change the sequences about once a week.
*   We added back a formal Curated Post section, mostly identical to the old one. Since longtime users often spend most time in the Community filter, Curated posts didn't stand out as strong as they used to. We had gotten some feedback that this made getting into Curated feel less special than it used to, and wanted to fix that.
*   Logged in users now always see all options for the "Curated/Frontpage/Community/Meta/Daily" menu
# Beyond algorithmic equivalence: self-modelling

In the [previous post](https://www.lesserwrong.com/posts/meG3Pai2YeRYcwPwS/beyond-algorithmic-equivalence-rewards), I discussed ways that the internal structure of an algorithm might, given the right [normative assumption](https://www.lesserwrong.com/posts/Fg83cD3M7dSpSaNFg/normative-assumptions-regret), allow us to distinguish bias from reward.

Here I'll be pushing the modelling a bit further.

Self-modelling
--------------

Consider the same toy anchoring biased problem as the previous post, with the human algorithm  H, some object  X, a random integer  0≤n≤99, and an anchoring bias given by

 H(X,n)=34V(X)+14n,

for  V some valuation function that is independent of  n.

On these inputs, the internal structure of  H is:

![](https://www.dropbox.com/s/izj2o2vxg9uswep/algorithm_model.png?raw=1)

However,  H is capable of self-modelling, to allow it to make long term decisions. At time  t,  H models itself at time  t+1 as:

![](https://www.dropbox.com/s/9g1vwop630dr5t1/model_models_model.png?raw=1)

Note that  H is in error here: it doesn't take into account the influence of  n on its own behaviour.

In this situation, it could be justifiable to say that  H's self model is the correct model of its own values. And, in that case, the anchoring bias can safely be dismissed as a bias.

Self-model and preparation
--------------------------

Let's make the previous setup a bit more complicated, and consider that, sometimes, the agent  H is aware of the effect of  n, and sometimes they aren't.

At time  t, they also have an extra action choice: either  n , which will block its future self from seeing  n, or  ∅, which will proceed as normal. Suppose further that whenever  H is aware of the effect of  n, they take action  n:

![](https://www.dropbox.com/s/9o1imzmnu8416x3/avoid_anchor.png?raw=1)

And when  H isn't aware of the effect of  n, they don't take any action/takes  ∅:

![](https://www.dropbox.com/s/4yptshicuc9toql/allow_anchor.png?raw=1)

Then it seems very justifiable to see  H as opposing the anchoring effect in themselves, and thus classifying it as a bias rather than a value/preference/reward.

The philosophical position
--------------------------

The examples in this post seem stronger than in the [previous one](https://www.lesserwrong.com/posts/meG3Pai2YeRYcwPwS/beyond-algorithmic-equivalence-rewards), in terms of justifying "the anchoring bias is actually a bias".

More importantly, there is a philosophical justification, not just an ad hoc one. We are assuming that < refresh to render LaTeX > has a self model of their own values - they have a model of what is a value and what is a bias in their own behaviour.

Then we can _**define**_ the reward of **< refresh to render LaTeX >** , **as the reward that < refresh to render LaTeX > models itself as having.**

In subsequent posts, I'll explore whether this definition is justified, how to access these self-models, and what can be done about errors and contradictions in self-models.
# Hammertime Intermission #2

The final cycle of Hammertime will return on 3/12.

This is an open thread for feedback about Hammertime.

Other Resources
---------------

Owen Shen has been writing about instrumental rationality for a long time at [MindLevelUp](https://mindlevelup.wordpress.com/). This material is much more coherent and actually cites the science behind the techniques.

SquirrelInHell has taken on a similar project but at a much deeper level at [BeWellTuned](http://bewelltuned.com/). These techniques are BLACK MAGIC powerful but may cost your soul. Consider Hammertime a gentle tutorial mission for BeWellTuned.

Epistemic Rationality
---------------------

A piece of the third cycle may be devoted to applying instrumental rationality tools towards increasing epistemic standards in practice.

What are the immediate applications of Hammertime tools towards truth-seeking? Are there TAPs, Design decisions, CoZE experiments, or Focusing techniques that you would immediately recommend to increase epistemic standards? For example, I’ve seen people claim that they’ve installed a gut-level instinct for Bayes Theorem.

How does truth-seeking interface with instrumental rationality, especially on corrupted hardware? We pay a whole lot of lip service to “I desire to believe what is true,” and yet an enormous amount of our practical advice sounds like “Act as if \[obviously false statement\] is true.”

Meta-Cognitive Blind Spots
--------------------------

I have a friend who got so excited about conquering new fears that he’s been constantly flitting around, looking for aversions to squash with CoZE. When I asked him to try sticking to one thing for a while, he nearly refused, saying “It feels hard to look at.” My words:

> when ppl get excited about CoZE

> there’s something ridiculous about it

> you’re literally looking for things that scare you

> like actually scare you

> and that’s not fun

One time this week, I was Focusing on the feeling of hating algebraic geometry homework. I found the feeling’s true name: “Working on problems in other fields makes me feel low-status.” The aversion completely vanished, but I was left with the sinking feeling that I’m status-seeking all the way down. In fact, this whole paragraph reporting this character flaw? It’s a self-deprecation status move too. Fuck.

_\[Commenting guidelines: At least one line of poetry per comment.\]_
# Monthly Meta: Common Knowledge

One of my favourite aspects of Less Wrong is that we've created a community where it is easier for us to communicate with each other because we have a shared understanding of certain concepts. A large amount of highly quality content has been posted to the site since it started, so I think it would be a useful exercise to reflect on which ideas should become part of this common knowledge for this edition of Monthly Meta.

I'd suggest that this should work as follows: each post should contain one concept, with a short definition and an explanation of why you think it is important. Please don't submit your own ideas. I'll go first.
# Interactive Bayes Theorem Visualization

This visualization was inspired by the Julia Galef [video](https://www.youtube.com/watch?v=BrK7X_XlGB8) that I linked to in the opening paragraph. One thing I liked about that video is that it represents probability space as a box which, for understanding Bayes' rule, I think is much more intuitive than the circular venn-diagram approaches. Some improvements that I think this visualization offers is showing the different probabilities updating in relation to each other, generalizing to more than two events for both priors and likelihoods, and explicitly showing the discarding of conditional probabilities of unobserved events and normalization of the posteriors.
# Funding for AI alignment research

If you are interested in working on AI alignment, and might do full or part time work given funding, consider submitting a short application to funding@ai-alignment.com.

Submitting an application is intended to be very cheap. In order to keep the evaluations cheap as well, my process is not going to be particularly fair and will focus on stuff that I can understand easily. I may have a follow-up discussion before making a decision, and I'll try not to favor applications that took more effort.

As long as you won't be offended by a cursory rejection, I encourage you to apply.

If there are features of this funding that make it unattractive, but there are other funding structures that could potentially cause you to work on AI alignment, I'm curious about that as well. Feel free to leave a comment or send an email to funding@ai-alignment.com (I probably won't respond, but it may influence my decisions in the future).
# Is there a Connection Between Greatness in Math and Philosophy?

I have nothing that looks like an answer, but I wanted to put the question out to start a discussion. I especially hope that this leads to someone here researching the question well. I also have almost nothing to back up the claims below.

I became curious about this from thinking about how to find people who can make progress in MIRI's Agent Foundations questions. It seems to require very high levels of both math and philosophy, and I want to know how to select for both of these skills simultaneously.

It seems like many of the great mathematicians at least had enough of an interest in philosophy to share their philosophical opinions. Also it appears that the people who substantial progress in philosophy often use math. (There are of course disagreements about what makes good philosophy, and I am biased in favor of the mathematical flavored philosophy.)

I am especially surprised by the fact that the tails seem to come together rather than apart. It seems like mediocre mathematicians and philosophers are much less likely to be interested in the other field.

Some possible theories:

1) I am noticing a pattern that is not there.

2) This is a consequence of the fact that all intelligence is correlated, and you will notice a similar pattern between many pairs of fields.

3) Mathematics and Philosophy are very old fields. Historically, fields did not really exist, and people were more interdisciplinary. If you look at modern mathematicians and philosophers, the trend goes away.

4) The skill to produce great math and skill to produce great philosophy are secretly the same thing. Many people in either field do not have this skill and are not interested in the other field, but the people who shape the fields do.

Thoughts?
# Beta-Beta – Recent Discussion

Over on our [beta-beta (I guess the technical term is "alpha") sit](https://www.lessestwrong.com/)e, we're experimenting with a new system for Recent Comment, renaming it to "Recent Discussion" and tweaking some features.

The recent comments are organized by post, sorted in the order of "post most recently commented on", showing the three most recent comments for each post.

If you haven't read a post, it also shows a short blurb of the first 300 characters of the post (similar to how the recent comments themselves do).

A couple motivations for this:

*   If there's one thread with loads of discussion happening, it can easily overwhelm the recent comments section. (For example, as of right now on Lesserwrong.com, all 10 Recent Comments are from the Jorden Peterson thread right now. If you haven't logged on in 4 hours, you'd have missed the discussion happening in Connection Between Math and Philosophy and Subtle Forms of Confirmation Bias. Whereas on LessestWrong it shows you each of those posts (plus a few others) clustered by thread.
*   High-volume discussions not only makes it harder to see all the active threads, but also produces runaway affects where "only one thread is showing up in Recent Comments, so people are predisposed to comment on that discussion instead of others."
*   Old posts that get commented on get highlighted slightly more than they were before (in particular, if you haven't read them already, you'll get a bit more context about what the post was about).
*   I wanted it to be easier to skim the opening text of new posts (empirically I don't actually click the "show highlight" button that I worked so hard on last month. :P). Up till now it's been way easier to peruse random comments with no context than to start reading actual posts, which seems backwards to me.
*   Posts with negative karma will not have their comments show up in the recent discussion, preventing runaway demon threads from taking over the front page.

This is a fairly major change, so we wanted to collect some feedback before moving forward on it, and it seemed easier to get that feedback if we temporarily connected the live database to the Lessestwrong.com. For the next few days you can log in there normally.

Longterm, there's a few other features that could get rolled into this, such as:

*   displaying proper threading for recent comments (since multiple comments from a single post can be loaded and once and sorted together)
*   highlight comments you haven't read
*   hide discussions on posts you aren't interested in

https://www.lessestwrong.com/
# [defunct] Moderation Action List (warnings and bans)

**Update: This hasn't been updated since 2021, and we're no longer updating it.**

This post is a (continually updated) list of moderation bans and warnings of LWers, with a link to the comment thread where they happened.

(Added: Note that the vast majority of bans are for brand new users who are norm-violating in their first comments or otherwise seem like bad user growth for their site, and they are not listed here. This list is for users who have had a fair bit of activity on the site.)

*   Dr. Jamchie is banned for three months for [reasons](https://www.lesswrong.com/posts/nbw2NfjiBmuGx7mim/ms-blue-meet-mr-green?commentId=vLhWELoSwKLynSkKT).
*   Tempus is banned for being [Eugine Nier](https://www.lesserwrong.com/posts/NGc3Yjecg9pDMznWq/moderator-action-eugine_nier-is-now-banned-for-mass-downvote).
*   diegocaleiro is banned for [reasons](https://www.lesserwrong.com/posts/zHiha9jcJweRAo9xk/the-sad-state-of-rationality-zuerich-effective-altruism#jXv6CGoXW8RvLdwB7).
*   PDV has had a [public](https://www.lesserwrong.com/posts/yPfwMoxyRX77DARjM/gnostic-rationality#RCc7d5QqNa4tKGHgZ) and [private](https://www.lesserwrong.com/posts/iTnyxwE8i9wNYh8D8/the-hamming-problem-of-group-rationality#FAE9Jmwtv2ub4T9rw) warning.
*   Steve Whetstone is banned for spamming lots of really long, crackpot-looking comments
*   Said Achmiz has had a [public](https://www.lesswrong.com/posts/i2Dnu9n7T3ZCcQPxm/zetetic-explanation#ZBSSpQkoLTNNEmTyb) warning.
*   Elo has had a [public](https://www.lesswrong.com/posts/4EGYhyyJXSnE7xJ9H/in-my-culture#AmcdwPHSxZSB7C492) warning.
*   ialdaboath is banned for [reasons](https://www.lesswrong.com/posts/uwBKaeQzsvkcErmBm/ialdabaoth-is-banned).
*   Gleb_Tsipursky is banned for [a long history of scammy behaviour](https://forum.effectivealtruism.org/posts/fn7bo8sYEHS3RPKQG/concerns-with-intentional-insights), including evidence that he paid MTurkers to upvote his LW comments (discussed in [this thread](https://www.lesswrong.com/posts/PAHRTzZWeN6gjLN3q/is-this-dark-arts-and-if-it-is-it-justified#9ck5JZsuEnDCjpJ5e), which offers [this recorded evidence](https://archive.ph/bJGcu)).
*   Curi is banned for [reasons](https://www.lesswrong.com/posts/PkpuvsFYr6yuYnppy/open-and-welcome-thread-september-2020?commentId=9AxhBKqNbc7tkwCAM)
*   Periergo is banned for [reasons](https://www.lesswrong.com/posts/PkpuvsFYr6yuYnppy/open-and-welcome-thread-september-2020?commentId=9AxhBKqNbc7tkwCAM)
*   IlyaShpitser has had a [public](https://www.lesswrong.com/posts/MnFqyPLqbiKL8nSR7/my-experience-at-and-around-miri-and-cfar-inspired-by-zoe?commentId=5ECKs3DrfTeNSwJXC) warning.
# Murphy’s Quest Ch 6: Perverse Incentives

How did I end up in this position?

Let’s review the facts.

First.

From Class Choice to ROT13, from Harry Potter villains to this chastity cult, from XP sharing to judgy old men, this world is out to get me.

Second.

I have despaired of living the Good Life and decided to take revenge.

Third.

There is a little-known law of biology which states that the male gaze, rapacious as it normally is, doubles in rapaciousness for each day ungratified. By my count, this makes me 32 times as susceptible to the female form as the average 16 year old male.

Fourth.

The only remotely feminine thing in this church is a life-sized Goddess statue in the chapel.

It is for these reasons that I find myself, just before the stroke of midnight on the 17th of February of the 2014th year of our Lord and Savior Jesus Christ, standing half-naked on the preacher’s podium in said chapel, facing said statue, relieving said ungratified urge.
# Is this what FAI outreach success looks like?

Highlights (spoilers):

Two hours of more or less reasonable discussion of AI safety.

Specifically, even though people disagree, they tend to keep arguments on the object level, rather than getting bogged down in how things sound or what people are allowed to talk about.

Plus, the Overton window for what gets included in the discussion is such that _Max Tegmark_ sounds like a _reasonable person_.

Neil deGrasse Tyson mentions that due to listening to a Sam Harris interview with an anonymous AI researcher, he has been convinced that we would not keep control over superintelligent AI by virtue of being able to unplug them.

The audience questions are by and large reasonable questions for people recently interested in the subject to be asking.

* * *

Admittedly, this is a stacked deck - Max is a known quantity, but the U of M professor, Mike Wellman, is also someone who's been doing research on safety and reliability topics. I'm reminded of Scott Alexander's (?) remark that in the course of AI safety "coming out of the closet," there's been a shift in what's acceptable to talk about without people actually changing their minds. The culture of what can be talked about has started to shift, but the panel still had to have been selected a little bit for people who had delved into the topic before (which makes me wonder, who selected the panel?).

If this is what success looks like, then success looks like

*   Shifting the culture of what's acceptable to talk about (through conferences, funding, talking to the media).
*   Broadening the circle of experts who have worked on this topic, are familiar with the research, and can serve as authorities and mentors.
*   Informing the public with media events such as this one.
*   Actually changing peoples' mind via reasonable arguments, at some nonzero rate (through websites, books, podcasts...).