Harley-ml/lesswrong huggingface dataset:
# Towards equilibria-breaking methods

All of this is vague intuition.

[Much](http://slatestarcodex.com/2014/07/30/meditations-on-moloch/) has been [said](https://www.lesswrong.com/posts/x5ASTMPKPowLKpLpZ/moloch-s-toolbox-1-2) about the problems of Nash equilibria. I've been wondering about how to quantify these sorts of problems. From Moloch's Toolbox:

> 3.  Systems that are broken in multiple places so that no one actor can make them better, even though, in principle, some magically _coordinated_ action could move to a new stable state.

My animating thought is that information and incentives aren't evenly distributed, and further there is an important time dimension to the problem. I expect it should be possible to temporarily disrupt any given equilibrium. From disruption, a better equilibrium could be reached. Alternatively, maybe it is possible to go directly to the better equilibrium.

*   We should be able to get knowledge of the actors.
*   Information, which is to say beliefs-about-the-current-equilibrium, is not perfect or even among them. _How_ imperfect and _how_ uneven?
*   With knowledge of the actors, we should be able to get a distribution of their incentives.
*   Now we know how many actors, and their incentives, and their information. With _this_ knowledge, we should be able to determine how big an incentive we need to offer in order to shift them into a target equilibrium.
*   If we want it to be a new equilibrium, we will need to affect enough actors to make it fairly stable. How many actors will we need, a la the [hundredth monkey effect?](https://en.wikipedia.org/wiki/Hundredth_monkey_effect)
*   It is probably important to be able to tailor incentives to actors.
*   My intuition for the threshold is that the weaker the belief in the current equilibrium, the less incentive will be required to shift them. So the target is something like (current incentives from equilibrium + incentive to overcome belief).

I think of this like surveying a rock face for blasting. Once we identify the lowest-dynamite method, anyone else with enough dynamite can come along and detonate.

My default assumption is that this will take the traditional form of getting a lot of capital and using that to directly or indirectly provide these incentives. Under this assumption there needs to be a reason for that capital to be available, which is to say a profit; maybe this is realizable through market operations directly. Alternatively this could be the vehicle for a startup's pitch; it seems like this process would qualify as superlative market research.

However, it does occur to me that actors might be more like _individuals_ who work in key positions, rather than entire firms. If that is the case I expect the belief-disparities to be even higher than they would be for firms, and the cost of shifting them probably smaller.

Near as I can tell, no one has really taken a startup-pitch-worth of effort to describe an attempt at shifting an equilibrium, especially not in the sense that we talk about them.
# Which textbook would you recommend to learn decision theory?

Eliezer talks a lot about decision theory in his sequences, e.g. the Aumann agreement theorem or the von Neumann-Morgenstern utility theorem. From what I've seen so far, decision theory looks extremely interesting.

Which textbook in decision theory would you recommend to start with? I'd appreciate if the book contained not only theory, but also some exercise/problem section - I have noticed that usually a lecture is not enough to fully grasp a topic. I want a book which will not shy away from mathematical side of the theory.

I have a strong background in mathematics and computer science, but I only know a little about game theory.
# Wireheading is in the eye of the beholder

tl;dr: there is no natural category called "wireheading", only wireheading relative to some desired ideal goal.

Suppose that we have a built an AI, and have invited a human H to help test it. The human H is supposed to press a button B if the AI seems to be behaving well. The AI's reward is entirely determined by whether H presses B or not.

So the AI manipulates or tricks H into pressing B. A clear case of the [AI wireheading itself](https://wiki.lesswrong.com/wiki/Wireheading).

Or is it? Suppose H was a meddlesome government inspector that we wanted to keep away from our research. Then we want H to press B, so we can get them our of our hair. In this case, the AI is *behaving entirely in accordance with our preferences*. There is no wireheading involved.

Same software, doing the same behaviour, and yet the first is wireheading and the second isn't. What gives?

Well, initially, it seemed that pressing the button was a proxy goal for our true goal, so manipulating H to press it was wireheading, since that wasn't what we intended. But in the second case, the proxy goal *is* the true goal, so maximising that proxy is not wireheading, it's efficiency. So it seems that the definition of wireheading is only relative to what we actually want to accomplish.

## In other domains

I similarly have the feeling that wireheading-style failures in [value-learning](https://agentfoundations.org/item?id=1099), low impact, and corrigibility, also depend on a specification of our values and preferences - or at least a partial specification. The more I dig into these areas, the more I'm convinced they require partial value specification in order to work - they are not fully value-agnostic.
# Applied Rationality podcast - feedback?

My goal is to help non-rationalists become rationalists, and provide exercises for existing rationalists to hone their skills, in the form of audio/video episodes.

'Rationally Speaking' and 'The Bayesian Conspiracy' are lovely, but they don't serve the function I have in mind.

Since I am an amature rationalist, and possibly the least intelligent person in this community, it would be ideal for someone else to run this (preferrably CFAR themselves, or at least someone who has attended).
If there is an existing project like this, please point me to it so I can support it.
If not, I at least have the drive to get the ball rolling.

I would probably start by using the 'Hammertime' sequence as material, and breaking it down a bit so the average layperson can understand it easier. This would require very little knowledge on my part, and give me time to maybe find a replacement host, cohost, or a queue of interviewees to assist in future episodes.

My question: is there any feedback you can give on this idea?
# Why is this utilitarian calculus wrong? Or is it?

Suppose that I value a widget at $30. Suppose that the widget costs the widget-manufacturer $20 to produce, but, due to monopoly power on their part, they can charge $100 per widget.

The economic calculus for this problem is as follows. $30 (widget valuation) - $100 (widget price) = -$70 to me; $100 (widget price) - $20 (widget cost) = $80 to widget producers. $80 - $70 = +$10 total value. Ordinarily, this wouldn't imply that utilitarians are required to spend all their money on widgets because for a function to convert dollars to utils u($), u'($)>0, u''($)<0 and widget-producers usually have higher $ then widget consumers.

But suppose the widget monopolist is a poor worker commune. The profits go directly to the workers who, on average, have lower $ then I do. It seems like buying widgets would be more moral then, say, donating $80 to the same group of poor people ($80 - $80 = $0) because the widget purchase slightly compensates me for the donation in a way that is greater then the cost of the recipient to produce the widget.

And yet, I feel even less moral compunction to buy widgets then I do to donate $80 to GiveDirectly. Is this just an arbitrary, unjustifiable, subconscious desire to shove economic transactions into a separate domain from charitable donations or is there actually some mistake in the utilitarian logic here? If there isn't a mistake in the logic, is this something that the Open Philanthropy Project should be looking at?

\[Question inspired by a similar question at the end of chapter 7 of Steven Landsburg's _The Armchair Economist_\]
# Rationality: What's the point?

_This post is part of my_ _[Hazardous Guide To Rationality.](https://www.lesswrong.com/s/pKxyvLyhEFxHgogdD)_ _I don't expect this to be new or exciting to frequent LW people, and I would super appreciate comments and feedback in light of intents for the sequence, as outlined in the above link._

A friend once articulated that he didn't like when things are taught, "Mr. Miygai style". A bunch of disconnected, unmotivated facts, exercises, and ideas are put before you, and it's only at the very end that it clicks and you see the hidden structure and purpose of everything you've learned.

Therefore, the very first post of this sequence is going to be a drive by of what I think some of the cool/useful/amazing things are that you can get out of The Way. I never would have become a close-up magician if I hadn't seen someone do incredible things that blew my mind.

Who Is This For?
================

As much as it pains me to say this, it might not really matter whether or not you follow The Way. It really depends on what you're trying to do. The guy who kicked of the contemporary rationality community, Eliezer Yudkowsky, notes that besides a natural draw based on personality, the biggest reason he's invested in rationality is because he really wants to make sure Friendly AI happens before Unfriendly AI, and turns out that's really hard.

\[\*add more\*\]

What's the Pot of Gold at the End of the Rainbow?
=================================================

Things I claim you can get better at

*   Believing true things and not believing false things.
*   Arrive at true beliefs faster.
*   "Failing mysteriously" less often
*   Understanding how your own mind works.

Why some of the above things are awesome

*   If you have something to protect, (you really want to make certain things happen) better models, more true beliefs, update speed, and being confused by lies, all make you more likely to make the changes you want to see in the world.
*   If you get a kick out of more deeply grokking how the world around you works, a kick you will get.
*   A lot of interpersonal problems come from two gaps:

*   One between "How human minds work" and "How you think human minds work"
*   One between "Your beliefs, feelings, and emotions" and "Your self-model of your beliefs, feelings, and emotions"
*   Shorting those gap will result in less interpersonal problems.
# What are some of bizarre theories based on anthropic reasoning?

Doomsday argument and simulation argument are quite bizarre for most people. But these are not the only strange theories one can come up with when employing anthropics.

Some examples:

**It is likely to have an unusual high IQ.**

Perhaps brain works in a way, that having high IQ correlates with something that also causes more observer moments. Hence there are more of high IQ experience in the world than that of low IQ.

**Fragile universe**

Total universe destroying physical catastrophes that expands in speed of light (say false vacuum collapse) could be very frequent. As much, as once every second. And it is only due to survivor bias that we think universe is stable and safe. How would we know?

**Animals does not have consciousness**

There are more animals than humans on Earth. Still we find ourselves as humans. Perhaps it is because we only can be humans, as only humans have consciousness.

**We are stuck inside infinite loop**

Lets assume simulation argument is correct. Then we probably exist inside simulation, run by some software. All software have bugs. One of the bugs software sometimes have is getting itself in an infinite loop. Biggest amount of experience computed by this software then could possibly be inside this infinite loop.
# AI Safety Prerequisites Course: Revamp and New Lessons

Previous post: [Fundamentals of Formalisation Level 7: Equivalence Relations and Orderings](https://www.lesswrong.com/posts/BkL78sXpNXuHcR64Q/fundamentals-of-formalisation-level-7-equivalence-relations). First post: [Fundamentals of Formalisation level 1: Basic Logic](https://www.lesswrong.com/posts/kK67yXhmDYwXLqXoQ/fundamentals-of-formalisation-level-1-basic-logic).

Nine months ago we, [RAISE](http://aisafety.info), have started creating a [Math Prerequisites for AI Safety online course](https://app.grasple.com/#/course/141?access_token=3HCK4oRipeFY2ghyYMqDJKYX57KUnzNL). It has mostly MIRI research related subjects: set theory, computability theory, and logic, but we want to add machine learning related subjects in the future. For 4 months we've been adding new lessons and announcing them on LessWrong. Then we stopped, looked back and decided to improve their usability. That's what we've been busy with since August.

***
![Diagram of levels in the prerequisites course](http://dump.bitcheese.net/files/kiratug/Screenshot_20190203_203007.png)

## News since the last post

1. Big update of 7 levels we had previously published, which you can see in the picture above. The lessons use textbooks, which you will need to follow along. Previously lessons looked like "read that section; now solve problems 1.2, 1.3, 1.4c from the textbook; now solve these additional problems we came up with". Now our lessons still say "read that section", but the problems (and their solutions, in contrast to many textbooks, which don't provide solutions) are included in lessons themselves. Additional problems are now optional, and we recommend that students skip them by default and do them only if they need more practice. New levels in Logic, Set Theory, and Computability tracks will be like that as well.
2. Level 1 was very long, consisted of 45 pages of reading, and could take 10 hours for someone unfamiliar with logic. We separated it into smaller parts.
3. Two new levels. Level 8.1: Proof by Induction. Level 8.2: Abacus Computability.

***

If you study using our course, please give us feedback. Leave a comment here or email us at raise@aisafety.camp, or through the [contact form](http://aisafety.camp). Do you have an idea about what prerequisites are most important for AI Safety research? Do you know an optimal way to learn them? Tell us using the same methods or collaborate with us.

Can you check if a mathematical proof is correct? Do you know how to make proofs understandable and easy to remember? Would you like to help to create the prerequisites course? If yes, consider [volunteering](http://aisafety.camp/study-group).
# If Rationality can be likened to a 'Martial Art', what would be the Forms?

In Martial Arts, we have 'forms' that allow us to practice our skills when not with a partner or in adversity.
In Rationality, I imagine that this would take the form of "brain teasers", but specifically regarding decision theory, overcoming biases, and calculating probabilities.

What are some things one can do to practice Applied Rationality? These can take the form of thought experiments, websites, apps, pen-and-paper tools, etc.

These would ideally range from "easy to do within a couple minutes" to "you need to dedicate a day to this".
# Thoughts on Ben Garfinkel's "How sure are we about this AI stuff?"

I liked [this talk by Ben](https://www.youtube.com/watch?v=E8PGcoLDjVk&fbclid=IwAR12bx55ySVUShFMbMSuEMFi1MGe0JeoEGe_Jh1YUaWtrI_kP49pJQTyT40).

I think it raises some very important points. OTTMH, I think the most important one is: **We have no good critics.** There is nobody I'm aware of who is seriously invested in knocking down AI-Xrisk arguments and qualified to do so. For many critics in machine learning (like Andrew Ng and Yann Lecun), the arguments seem obviously wrong or misguided, and so they do not think it's worth their time to engage beyond stating that.

A related point which is also important is: **We need to clarify and strengthen the case for AI-Xrisk.** Personally, I think I have a very good internal map of the path arguments about AI-Xrisk can take, and the type of objections one encounters. It would be good to have this as some form of flow-chart. **Let me know if you're interested in helping make one.**

Regarding machine learning, I think he made some very good points about how the the way ML works doesn't fit with the paperclip story. I think **it's worth exploring the disanalogies more and seeing how that affects various Xrisk arguments.**

As I reflect on what's missing from the conversation, I always feel the need to make sure it hasn't been covered in _Superintelligence_. When I read it several years ago, I found _Superintelligence_ to be remarkably thorough. For example, I'd like to point out that FOOM isn't necessary for a unilateral AI-takeover, since an AI could be progressing gradually in a box, and then break out of the box already superintelligent; I don't remember if Bostrom discussed that.

The point about **justification drift** is quite apt. For instance, I think the case for MIRI's veiwpoint increasingly relies on:

1) optimization daemons (aka "inner optimizers")

2) adversarial examples (i.e. current ML systems seem to learn superficially similar but deeply flawed versions of our concepts)

TBC, I think these are quite good arguments, and I personally feel like I've come to appreciate them much more as well over the last several years. But I consider them far from conclusive, due to our current lack of knowledge/understanding.

**One thing I didn't quite agree with in the talk:** I think he makes a fairly general case against trying to impact the far future. I think the magnitude of impact and uncertainty we have about the direction of impact mostly cancel each other out, so even if we are highly uncertain about what effects our actions will have, it's often still worth making guesses and using them to inform our decisions. He basically acknowledges this.
# My use of the phrase "Super-Human Feedback"


I've taken to calling Debate, Amplification, and Recursive Reward Modeling **"Super-human feedback" (SHF)** techniques. The point of this post is just to introduce that terminology and explain a bit why I like it and what I mean by it.

By calling something SHF I mean that it aims to outperform a single, unaided human H at the task of providing feedback about H's intentions for training an AI system. I like thinking of it this way, because I think it makes it clear that these three approaches are naturally grouped together like this, and might inspire us to consider what else could fall into that category (a simple example is just using a team of humans).

I think this is very similar to "scalable oversight" (as discussed in Concrete Problems), but maybe different because:

1) It doesn't imply that the approach must be scalable

2) It doesn't require that feedback is expensive, i.e. it applies to things where human feedback is cheap, but we can do better than the cheap human feedback with SHF.
# Open Thread February 2019

If it’s worth saying, but not worth its own post, you can put it here.

Also, if you are new to LessWrong and want to introduce yourself, this is the place to do it. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome. If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), and [seeing if there are any meetups in your area](https://www.lesswrong.com/community).

The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# What is learning?

I want to know what learning actually _is._ Ideally you can provide a model that is compact and useful as possible, such that when I encounter the word 'learn' in daily life I could replace it with this model while seeing a complete set of unique gears within it such that I can look at a real-world learning system and see how its traits correspond to each of those parts.

Generative subquestions:

What results from [Tabooing](https://www.lesswrong.com/posts/WBdvyyHLdxZSAMmoz/taboo-your-words) it? What are its components? Is there a _precise_ General Theory of Learning that underpins humans, animals, ML agents, or any other things that learn? If there are multiple constructs being pointed to with 'learn': what are the _differences_ between systems that 'learn'? How many different kinds of 'learning' are there and what do they look like?

What _isn't_ learning?
# When should we expect the education bubble to pop? How can we short it? 

I won't attempt to summarise the case for there being an education bubble here (see links below for some pointers). Rather, my questions are:

1) _assuming_ there is an education bubble, when will it -- as bubbles tend to do -- pop?

(This plausibly entails some disjunction of \*hundreds of thousands to millions of students defaulting on their debt, \*lower number of college applicants, \*non-top-tier colleges laying off faculty, \*substantial reductions the signalling value of obtaining a diploma, \*substantial reductions in tuition fees, \*reduction in the level of education required by various employers, and more)

2) Which assets will be more scarce/in demand as that happens? Are there currently available opportunities for "shorting" the education bubble and invest in ways which will yield profit when it pops?

(I hereby preface the comments by noting that nothing discussed there is investment advice and no users can be held liable for investment decisions based on it.)

* * *

[Peter Thiel summarises the inside view of there being an "education bubble" well](https://www.youtube.com/watch?v=crAHDXdBCXg).

And here are some interesting numbers:

*   In the last 35 years, median household income has grown by about 20% ([FRED](https://fred.stlouisfed.org/series/MEHOINUSA672N)). In roughly the same time, [the price of college has grown by 300% when adjusting for inflation](https://www.lesswrong.com/posts/Fafzj3wMvoCW4WjeF/against-tulip-subsidies)
*   [College spending is one sixth of US economy](http://fortune.com/2017/07/10/higher-education-student-loans-economic-bubble-federal/) UPDATE: this is probably false/misleading, see comment from paulfchristiano below.
*   [Student debt is at >$1 trillion](https://www.forbes.com/sites/zackfriedman/2017/02/21/student-loan-debt-statistics-2017/#2ca68b5d5dab) (for comparison US GDP is around $20 trillion, US federal budget is around $4 trillion)
# The Argument from Philosophical Difficulty

(I'm reposting [this comment](https://www.lesswrong.com/posts/JbcWQCxKWn3y49bNB/disentangling-arguments-for-the-importance-of-ai-safety#daD7JREPtx2WDe2Wf) as a top-level post, for ease of future reference. The [context](https://www.lesswrong.com/posts/JbcWQCxKWn3y49bNB/disentangling-arguments-for-the-importance-of-ai-safety) here is a discussion about the different lines of arguments for the importance of AI safety.)

Here's another argument that I've been pushing since the [early days](http://www.sl4.org/archive/0711/17101.html) (apparently not very successfully since it didn't make it to this list :) which might be called "argument from philosophical difficulty". It appears that achieving a good long term future requires getting a lot of philosophical questions right that are hard for us to answer. Given this, [initially](https://www.lesswrong.com/posts/vrnhfGuYTww3fKhAM/three-approaches-to-friendliness) I thought there are only three ways for AI to go right in this regard (assuming everything else goes well with the AI):

1. We solve all the important philosophical problems ahead of time and program the solutions into the AI.

2. We solve metaphilosophy (i.e., understand philosophical reasoning as well as we understand mathematical reasoning) and program that into the AI so it can solve philosophical problems on its own.

3. We program the AI to learn philosophical reasoning from humans or use human simulations to solve philosophical problems.

Since then people have come up with a couple more scenarios (which did make me *slightly* more optimistic about this problem):

4. We all coordinate to stop technological progress some time after AI but before space colonization, and have a period of long reflection where humans, maybe with help from AIs, spend thousands or millions of years to solve philosophical problems.

5. We program AIs to be corrigible to their users, some users care about getting philosophy correct so the AIs help keep them safe and get their "fair share" of the universe until philosophical problems are solved eventually, enough users care about this so that we end up with a mostly good future, and lack of philosophical knowledge doesn't cause disaster in the meantime. (My writings on "human safety problems" were in part a response to this suggestion, outlining how hard it would be to keep humans "safe" in this scenario.)

The overall argument is that, given [human safety problems](https://www.lesswrong.com/posts/HTgakSs6JpnogD6c2/two-neglected-problems-in-human-ai-safety), realistic competitive pressures, difficulties with coordination, etc., it seems hard to end up in any of these scenarios and not have something go wrong along the way. Maybe another way to put this is, given philosophical difficulties, the target we'd have to hit with AI is even smaller than it might otherwise appear.
# Probability space has 2 metrics

A metric is technically defined as a function from pairs of points to the non negitive reals.  d:X×X→\[0,∞) With the properties that d(x,y)=d(y,x) and d(x,y)=0⟺x=y and d(x,y)+d(y,z)≥d(x,z).

Intuitively, a metric is a way of measuring how similar points are. Which points are nearby which others. Probabilities can be represented in several different ways, including the standard p∈(0,1) range and the log odds b∈(−∞,∞). They are related by b=log(p1−p) and eb−1=11−p and p=ebeb+1 (equations algebraically equivalent)

The two metrics of importance are the baysian metric B and the probability metric P.

B(b1,b2)=|b1−b2|=∣∣ ∣∣log(p1(1−p2)p2(1−p1))∣∣ ∣∣

P(p1,p2)=|p1−p2|=∣∣∣1eb1+1−1eb2+1∣∣∣

Suppose you have a prior, b1 in log odds, for some proposition. Suppose you update on some evidence that is twice as likely to appear if the proposition is true, to get a posterior, b2 in log odds. Then B(b1,b2)=log(2). The metric B measures how much evidence you need to move between probabilities.

Suppose you have a choice of actions, the first action will make an event of utility u happen with probability p1, the other will cause the probability of the event to be p2. How much should you care. uP(p1,p2).

The first metric stretches probabilities near 0 or 1 and is uniform in log odds. The second squashes all log odds with large absolute value together, and is uniform in probabilities. The first is used for baysian updates, the second for expected utility calculations.

Suppose an imperfect agent reasoned using a single metric, something in between these two. Some metric function less squashed up than P but more squashed than B around the ends. Suppose it crudely substituted this new metric into its reasoning processes whenever one of the other two metrics was required.

In decision theory problems, such an agent would rate small differences in probability as more important than they really were when facing probabilities near 0 or 1. From the inside, the difference between no chance and 0.01, would feel far larger than the distance between probabilities 0.46 and 0.47.

[The Allais Paradox](https://www.lesswrong.com/posts/zJZvoiwydJ5zvzTHK/the-allais-paradox)

However, the metric is more squashed than B, so moving from a 10000:1 odds to 1000:1 odds seems to require less evidence than moving from 10:1 to 1:1. When facing small probabilities, such an agent would perform larger baysian updates than really necessary, based on weak evidence.

[Privileging the Hypothesis](https://www.lesswrong.com/posts/X2AD2LgtKgkRNPj2a/privileging-the-hypothesis)

As both of these behaviors correspond to known human biases, could humans be using only a single metric on probability space?
# Structured Concurrency Cross-language Forum

There are structured-concurrency-related efforts going on for different programming languages but the entire effort is kind of scattered, without people being aware of each other and speaking to each other.

If we had a common forum, we could share the use cases, the problems, the ideas and the solutions. Each of us, irrespective of which language they are working with, could benefit from this common pool of knowledge…

And here we go!

The forum [now exists](https://trio.discourse.group/c/structured-concurrency). Thanks to Nathaniel Smith for setting it up!

And here's my [kick-off post on the forum](https://trio.discourse.group/t/structured-concurrency-kickoff).

Enjoy!

**Feb 10th, 2019**
# Why do you reject negative utilitarianism?

_Note (2022-03-15): For a recent defense of similar axiological views, see my [series on minimalist axiologies](https://forum.effectivealtruism.org/s/MBadsrYLmzLNmYjaj)._

* * *

(Crossposted on the [EA Forum](https://forum.effectivealtruism.org/posts/Wj8JymeiMN28Hrrjo/why-do-you-reject-negative-utilitarianism))

Absolute negative utilitarianism (ANU) is a minority view despite the theoretical advantages of terminal value monism (suffering is the only thing that motivates us “by itself”) over pluralism (there are many such things). Notably, ANU doesn’t require solving value incommensurability, because all other values can be instrumentally evaluated by their relationship to the suffering of sentient beings, using only one terminal value-grounded common currency for everything.

Therefore, **it is a straw man argument that NUs don’t value life or positive states**, because NUs value them instrumentally, which may translate into substantial practical efforts to protect them (compared even with someone who claims to be terminally motivated by them).

If the rationality and EA communities are looking for a unified theory of value, why are they not converging (more) on negative utilitarianism?

What have you read about it that has caused you to stop considering it, or to overlook it from the start?

Can you teach me how to see positive states as terminally (and not just instrumentally) valuable, if I currently don’t? (I still enjoy things, being closer to the extreme of hyperthymia than anhedonia. Am I platonically blind to the intrinsic aspect of positivity?)

And if someone wants to answer: What is the most extreme form of suffering that you’ve experienced and believe can be “outweighed” by positive experiences?
# Would I think for ten thousand years?

Some AI safety ideas delegate key decisions to our idealised selves. This is sometimes phrased as "allowing versions of yourself to think for ten thousand years", or similar sentiments.

Occasionally, when I've objected to these ideas, it's been pointed out that any attempt to construct a safe AI design would involve a lot of thinking, so therefore there can't be anything wrong with delegating this thinking to an algorithm or an algorithmic version of myself.

But there is a tension between "more thinking" in the sense of "solve specific problems" and in the sense of "[change your own values](https://www.lesswrong.com/posts/vgFvnr7FefZ3s3tHp/mahatma-armstrong-ceved-to-death)".

An unrestricted "do whatever a copy of Stuart Armstrong would have done after he thought about morality for ten thousand years" seems to positively beg for [value drift](https://en.wikipedia.org/wiki/Instrumental_convergence#cite_note-13) (worsened by the difficulty in defining what we mean by "a copy of Stuart Armstrong [...] thought [...] for ten thousand years").

A more narrow "have ten copies of Stuart think about these ten theorems for a subjective week each and give me a proof or counter-example" seems much safer.

In between those two extremes, how do we assess the degree of value drift and its potential importance to the question being asked? Ideally, we'd have a [theory of human values](https://www.lesswrong.com/posts/zvrZi95EHqJPxdgps/why-we-need-a-theory-of-human-values) to help distinguish the cases. Even without that, we can use some common sense on issues like length of thought, nature of problem, bandwidth of output, and so on.
# Individual profit-sharing?

Here's a sketch of an idea:

*   Design an open-source legal agreement that two people sign.
*   The contract states that each person agrees to give the other 1% of their annual earnings, each year for _X_ years. (Ideally _X_ = several decades; both duration & percentage could be customized)
*   Contract is legally binding; each year both parties pay out to each other.
*   Not exclusive: a person could be in multiple contracts simultaneously (e.g. 5 contracts with 5 friends, sharing in total 5% of their annual earnings).

* * *

Two motivations for signing a contract like this:

1.  Diversify one's career & earnings risk by "investing" in admired peers.
2.  Deepen one's relationship with the other signee (signing isn't a thing to be taken lightly); signing signals intimacy & desire to build a longterm relationship with the other person.

* * *

Of course there are lots of ways something like this could go awry.

Has anyone heard of people doing something like this?

What are existing mechanisms that do something like this? Examples I've encountered already include marriage (50% profit-sharing indefinitely, at least in the US) and Kibbutzim (100% profit-sharing during one's tour of duty).
# Who owns OpenAI's new language model?

OpenAI published about [their very general language-writing & comprehension model](https://blog.openai.com/better-language-models/) ([a](https://web.archive.org/web/20190214171619/https://blog.openai.com/better-language-models/)) today.

Interestingly, they're not releasing the model itself:

> Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT-2 along with sampling code. We are not releasing the dataset, training code, or GPT-2 model weights.

Does OpenAI hold a copyright for this model (and its other intellectual property)?

If the US government decided they wanted to requisition this model, would an OpenAI copyright defend against that? Would OpenAI have any real recourse against a government requisition?
# How does OpenAI's language model affect our AI timeline estimates?

OpenAI [recently announced](https://blog.openai.com/better-language-models/) progress in NLP, using a large transformer-based language model to tackle a variety of tasks and breaking performance records in many of them. It also generates synthetic short stories, which are surprisingly good.

How surprising are these results, given past models of how difficult language learning was and how far AI had progressed? Should we be significantly updating our estimates of AI timelines?
# So you want to be a wizard

This is a link post to a recent talk by Julia Evans, popular programming blogger, called "So you want to be a wizard". I'm linking it on LW because I think it's one of the better takes I've seen on how to "apply rationality" to programming. A few interesting examples of things she discusses that I think will appeal to the LW audience:
- How she [built a game](https://computers-are-fast.github.io/) to get better at Fermi estimate of how long different computer operations should take.
- How believing that she can achieve a gears-level understanding of things she doesn't yet know much about makes her a better programmer and debugger.
- How she uses "premortem"s to predict what will go wrong with projects.

While I suspect none of the above will be new to much of the LW audience, I think she does a good job of connecting each strategy she discusses to real-world examples that help you understand what it would actually look like to apply them.
# Some disjunctive reasons for urgency on AI risk

(This has been sitting in my drafts folder since August 2017. Robin Hanson's recent [How Lumpy AI Services?](https://www.overcomingbias.com/2019/02/how-lumpy-ai-services.html) made me think of it again. I'm not sure why I didn't post it back then. I may have wanted to add more reasons, details and/or citations, but at this point it seems better to just post it as is. Apologies to those who may have come up with some of these arguments earlier.)

Robin Hanson recently [wrote](http://www.overcomingbias.com/2017/08/foom-justifies-ai-risk-efforts-now.html), "Recently AI risk has become something of an industry, with far more  going on than I can keep track of. Many call working on it one of the  most effectively altruistic things one can possibly do. But I&rsquo;ve  searched a bit and as far as I can tell that foom scenario is still the  main reason for society to be concerned about AI risk now." (By "foom scenario" he means a local intelligence explosion where a single AI takes over the world.) In response, I list the following additional reasons to work urgently on AI alignment.

1. Property rights are likely to not hold up in the face of large capability differentials between humans and AIs, so even if the intelligence explosion is likely global as opposed to local, that doesn't much reduce the urgency of working on AI alignment.

2. Making sure an AI has aligned values and strong controls against value drift is an extra constraint on the AI design process. This constraint  appears likely to be very costly at both design and run time, so if the  first human level AIs deployed aren't value aligned, it seems very  difficult for aligned AIs to catch up and become competitive.

3. AIs' control of the economy will grow over time. This may happen slowly  in their time frame but quickly in ours, leaving little time to solve value alignment problems before human values are left with a very small share of the universe, even if property rights hold up.

4. Once we have human-level AIs and it's really obvious that value alignment is difficult, superintelligent AIs may not be far behind.  Superintelligent AIs can probably find ways to bend people's beliefs and  values to their benefit (e.g., create highly effective forms of  propaganda, cults, philosophical arguments, and the like). Without an equally capable, value-aligned AI to protect me, even if my property rights are technically secure, I don't know how I would secure my mind.
# Major Donation: Long Term Future Fund Application Extended 1 Week

(Crossposted to the [EA Forum](https://forum.effectivealtruism.org/posts/5G5xspvM8wro78Gds/major-donation-ltf-ea-fund-application-extended-1-week))

[The Long-Term Future EA Funds](https://app.effectivealtruism.org/funds/far-future) application round is **extended for one more week (**until Midnight (PST) Sunday 24th of February).

The LTF Fund received a major donation such that **the amount of money available for granting has more than doubled, to a total of ~$1.2 million**, making our available funds greater than all money that the LTF fund has ever given away historically.

This means we can consider:

*   Grants of much larger ambition and scope.
*   Grants with longer time-horizons (who will need funding in future years)
*   Grants to larger teams
*   Seed funding for new organisations
*   Many more grants to smaller projects and individuals

I think this means applying this round is a particularly good idea, as well as forwarding the application to anyone else you know who might want to apply.

**[Apply here.](https://docs.google.com/forms/d/e/1FAIpQLSeDTbCDbnIN11vcgHM3DKq6M0cZ3itAy5GIPK17uvTXcz8ZFA/viewform?usp=sf_link)**

* * *

More info on the LTF Fund can be found [here](https://app.effectivealtruism.org/funds/far-future), you can see our last round of grants [here](https://app.effectivealtruism.org/funds/far-future/payouts/3JnNTzhJQsu4yQAYcKceSi).

If you’re uncertain about whether we’re interested in funding a project, you can ask me in a comment, or email me and the other fund managers at ealongtermfuture@gmail.com.
# Implications of GPT-2

I was impressed by [GPT-2](https://blog.openai.com/better-language-models/), to the point where I wouldn't be surprised if a future version of it could be used [pivotally](https://arbital.com/p/pivotal/) using existing protocols.

Consider generating half of a [Turing test](https://en.wikipedia.org/wiki/Turing_test) transcript, the other half being supplied by a human judge. If this passes, we could immediately implement an [HCH](https://ai-alignment.com/strong-hch-bedb0dc08d4e) of AI safety researchers solving the problem if it's within our reach at all. (Note that training the model takes much more compute than generating text.)

This might not be the first pivotal application of language models that becomes possible as they get stronger.

It's a source of superintelligence that doesn't automatically run into utility maximizers. It sure doesn't look like AI services, lumpy or no.
# A Strange Situation

Hello all,

I am a newcomer to the site, but I have been lurking and reading the various posts on this site for about 8 months. I have to tell you, life has started making a lot more sense since I read these posts. However, I am feeling somewhat directionless after reading the archives, as I cannot determine a next course of action.

The difficulty is this: I assume most of the poeple on this site are at least midle-aged; I had just turned 18 when I started browsing the site, after being vaguely dissatisfied with Rationalwiki's ability to keep me informed.

Given that, would anyone recommend a general path to follow in terms of what I should be reading from here? I have largely perused Yudkowsky's more well-known works, and have made it partially through the Codex.

Any help, recommendations, or suggestions would be appreciated, and I look forward to new horizons of knowledge as I may find them.
# Kocherga's leaflet

Some context: Kocherga is a [rationalist community hub in Moscow](https://www.lesswrong.com/posts/WmfapdnpFfHWzkdXY/rationalist-community-hub-in-moscow-3-years-retrospective).

There was a large crowdfunding campaign for printing the Russian HPMoR edition last year. Kocherga was the main local pickup point in Moscow; we distributed around 5000 copies to 3000 people.

We wanted to promote Kocherga, rationality and EA among these people (most of them just know HPMoR and don't know much about LW/EA), so I designed a new tri-fold leaflet. We handed it out with each set of books.

The reaction to this leaflet was quite positive, so I thought it'd be useful to do an English translation of it and share it here:

[![Page 1](https://berekuk.ru/files/tri-fold-2018-en.png)](https://berekuk.ru/files/tri-fold-2018-en.png)

[![Page 2](https://berekuk.ru/files/tri-fold-2018-en2.png)](https://berekuk.ru/files/tri-fold-2018-en2.png)

Some notes:

*   Capital letters in "What's happening in Kocherga" list read "cthulhu fhtagn" in Russian, but I wasn't able to adapt it to English. This might explain why the list looks kinda disorganized.
*   The translation is quite crude, it took me an hour and I didn't edit it as much as I could. If the tone feels a bit off, that's probably the reason.
*   The labels on the Dunning-Kruger graph are «Competence» on x axis and «Test results» on y axis.
* Here's an photo of the actual leaflet with HPMoR books: [https://www.instagram.com/p/BqC7eQfl7T7/](https://www.instagram.com/p/BqC7eQfl7T7/)
*   If you _really_ like the leaflet and would like to adapt it for your local community, let me know and I'll share the InDesign source file with you.

PS: We're still financially troubled, so if you like what we're doing and want to make sure we keep doing it, please consider [supporting Kocherga on Patreon](https://patreon.com/kocherga).
# [Link] OpenAI on why we need social scientists

[https://distill.pub/2019/safety-needs-social-scientists/](https://distill.pub/2019/safety-needs-social-scientists/)

> We believe the AI safety community needs to invest research effort in the human side of AI alignment. Many of the uncertainties involved are empirical, and can only be answered by experiment. They relate to the psychology of human rationality, emotion, and biases. Critically, we believe investigations into how people interact with AI alignment algorithms should not be held back by the limitations of existing machine learning. Current AI safety research is often limited to simple tasks in video games, robotics, or gridworlds, but problems on the human side may only appear in more realistic scenarios such as natural language discussion of value-laden questions. This is particularly important since many aspects of AI alignment change as ML systems [increase in capability](https://distill.pub/2019/safety-needs-social-scientists/#harder).
# Impact Prizes as an alternative to Certificates of Impact

**I posted this one to the EA Forum, because it seemed a bit more relevant there. I think that LessWrong users will also find it interesting.**

*TLDR Example*

An EA donor puts up a $50k prize for distribution in 2022. In 2022, several projects that have started since 2019 apply. Their net EA impacts are estimated, and these estimates (vs. the total value estimate of all submissions) are eventually used to give them corresponding proportional amounts of the $50k.

Back in 2019, several projects sell “rights” to their prize, and these get sold around. It’s expected that $1M in estimated total value will apply, so the market value of the claim of every $10 of estimated impact is ​$0.50. One project sets up an estimation service where they publicly estimate the eventual evaluation of every project, to help make the market more efficient, with the goal of themselves getting part of the prize.
# Is there an assurance-contract website in work?

I bet I'm not the only one who, after finishing inadequate equilibria, thought in excitement "Ok, so where's the KickStarter for better Nash equlibria?".

I didn't find any existing site that does something like it, but i wonder if maybe someone in this community is working on it.

If you know anything about it, I'd appreciate a hint ;)
# When does introspection avoid the pitfalls of rumination?

A theme I see being explored in the LW-sphere is the importance of connecting to system 1 information in various ways.

There's lots of stuff about meditation, [Focusing](https://medium.com/@ThingMaker/focusing-for-skeptics-6b949ef33a4f), [connecting to sub-agents](https://www.lesswrong.com/posts/5gfqG3Xcopscta3st/building-up-to-an-internal-family-systems-model), the question ["What would happen if [...] you introspected instead?"](https://www.lesswrong.com/posts/uQTXikjByTtvAR3r9/silence).

On the other hand, it appears to be received wisdom that a collection of behaviours called [rumination](https://en.wikipedia.org/wiki/Rumination_(psychology)) are maladaptive. But "focused attention on the symptoms of one's distress, and on its possible causes and consequences, as opposed to its solutions" doesn't sound a million miles from some of the techniques in the paragraph above.

What gives? Are some of the techniques bad? Is rumination OK? Are they importantly different, and if so, how?